Filippo Zanardo's Blog

to blog or not to blog

XenServer Final Backup Script with or without snapshot

with 62 comments

Ok guys,
Xenserver 5.6 is out now, and i enhance my backup script i rewrite some part and add some more features/controls, that i will explain later.

UPDATED: 2.2 is now avaiable i rewrite the check avaiable space to fix a minor bug

You can donwload the 2.1 script here

http://xenbackup.googlecode.com/files/xenbackup2.1.zip

It’s simple to set up just modify config.txt and than you can run the script with this simple command line:

perl /path/to/backup.pl /path/to/config.txt

Now i will explain what you can configure and the features:

@skip: this variable contain the uuid of the machine to skip in the backup you can get with the command ‘xe vm-list’, add more uuid separeted by ‘,’

usesnapshot: if set to true backup script try to make a snapshot of the vm, else he shutdown the machine, export and power on the machine

removable: if set to true the script detach the removable storage, else snapshot fail, and then reattached in the correct order, just remember to set up the variable removableuuid with your removable uuid you can get with command ‘xe sr-list’

mailNotification: if set to true script send email to and from the email specified in the config file, i remove the script used before now script use ssmtp that is already installed in xenserver, just configure the file /etc/ssmtp/ssmtp.conf with your data and remember to uncomment this line FromLineOverride=YES

subfolder: Set to true to create a subfolder in the store for each backup based on vm name

versioning: Set to true to let the script manage to delete the backup older than a certain day or number or hours specified in the $delnumber variable

logging: if set to true the script write all the message with data into one log/file specified in the variable $logfile

automount: if set to true script try to mount the backupdir specified in mountcommand at start and umount at end, else no action taken and u have to mount dir manually

checkspace: if set to true the script check the avaiable space on the backup dir and if less than $spacerequired quit with a message, size is in MB

i add also some check to verify that the correct file is been written on the backup folder, if not backup error is notified.

That’s all i hope i explain well and you like the script, comment or post for bugs, feature etc etc.

Cheers!

Advertisements

62 Responses

Subscribe to comments with RSS.

  1. […] with 17 comments DEPRECATED Check this new better version of the script […]

  2. This is working fantastic for me… Thank you!!
    One thing I have noticed is that after a few times of backing up, even though the script deletes snapshots, disk space keeps dropping pretty rapidly. I’ve ready somewhere else about xenserver doing that… is there a way to free up that lost space or is that something I could add into the script for it to do on completion of backups?

    Chris

    June 20, 2010 at 17:55

    • Hi glad to hear that works 🙂
      Btw u can check this link and try to coalescence-leaf (reclamining disk space), i didn’t add to the script cause running this script will incur some downtime for the VM, due the suspend/resume operations performed, and so i prefer to let people do manually.

      check this

      cheers

      pipposan

      June 21, 2010 at 10:15

  3. I can’t seem to get the notification portion working. The command will work fine from command line (ssmtp emailaddress < /tmp/emailmessage) but it won't run from the script even though it's definitely backing up ok. Any ideas?

    Josh

    June 28, 2010 at 19:36

    • hi,
      can u check in /var/log/maillog if there is any mail sent in the log? or post the last line. maybe you’re backing up you’re dns server? and so the machine can’t resolv outside.

      pipposan

      June 29, 2010 at 11:48

      • The ssmtp command from backup.pl works fine if I run it from the command line, it just doesn’t run with the script from cron.

        Here’s the /var/log/maillog

        Jun 30 04:40:40 domain-vm01 sSMTP[26997]: Sent mail for root@domain.com (221 2.0.0 Service closing transmission channel) uid=0 username=root outbytes=575

        From that it looks like it sent mail at 4:40am (the time the backup finishes) but yet I don’t receive it. One of the VM’s being backed up is the DNS server, but I don’t see how that matters?

        Josh

        June 30, 2010 at 21:27

      • hi, mmm strange if the system deliver the mail u should get it. btw if u shutdown and not snapshot the dns maybe the system can’t resolv outside address but it’s not your case watching the log.
        make a test please, put all your machine uuid in the skip variable and rerun the script so no backup is taken but we see if the mail will be sent correctly.
        and post back.
        cheers

        pipposan

        July 1, 2010 at 09:20

      • Sorry for taking so long to reply, I was on vacation.

        When adding all but 1 VM to the list of skips and running perl ~/backup.pl ~/config.txt the script emails fine at the end. So something about the cron job is stopping it from emailing or stops the process from running nightly.

        What should I check next?

        Josh

        July 12, 2010 at 18:58

      • mmm strange repost the /var/log/maillog and check if the script try to send the email, next can be spam filter, or something else that we can try to discover.

        pipposan

        July 13, 2010 at 17:45

      • Yes, the script is trying to send the email fine. It’s just not going out. The script makes it look like any other attempted send except for the fact that when I manually run the script it says xenbackup@ as the email address and during the cronjob run it says root@ as the email address. Maybe that’s the whole problem. This isn’t a spam filter or anything else, it’s not leaving the box. When I run it manually it works perfectly and emails me everytime, whether it’s 0 VM’s backed up, or 10. What would you suggest I check?

        Josh

        July 16, 2010 at 16:58

      • mmm actually is very strange, maybe it’s a cron problem, try to schedule a test backup with at, and let’s see what happens, or modify the /etc/crontab file to run when u can monitor the execution, and tail your mylog and maillog.
        ex for at command: “at -f mytest.sh -v 10:25”

        pipposan

        July 20, 2010 at 14:02

      • Hi Josh and pipposan
        Did you ever solve the problem with the email notifications which didn’t go out?
        I’ve the same problem: In the /var/log/maillog I see that a message was sent with a wrong sender address and it seems also that there was a wrong input file linked with the ssmtp command. If I send /tmp/emailmessage manually after the job there is an other message size in the maillog. The problem seems to be only when the script was started by a cron job.
        Any ideas?

        Patrik

        January 25, 2011 at 16:33

      • I have no problem with the mail, as long as you set the ssmtp, btw can u post some line of /var/log/maillog so i can try to help you.
        cheers

        pipposan

        January 29, 2011 at 14:24

      • Thanks for your response.
        In the meantime I did some tests and was able to solve the problem:
        After I’ve added the full path to ssmtp (/usr/sbin/ssmtp) at the corresponding lines in the script the emails are also correctly sended when the script is called by cron.

        Patrik

        January 29, 2011 at 15:35

  4. I dont know what I´m doing wrong.
    I have space to snapshot and on my cifs share.

    Tue Aug 3 16:16:18 2010 Executing: mount -t cifs //172.20.15.247/backup /vms/ -o username:*****,password=******
    Tue Aug 3 16:16:18 2010 Backup Error No space avaiable for the backup
    Tue Aug 3 16:16:18 2010 Executing: umount /vms/

    Could you help me please?

    Tks

    Marcos

    August 3, 2010 at 21:20

    • are u using the 2.1? cause 2.0 have a little bug that sometimes script seems to fail space avaiable check that is been fixed on 2.1.
      Try the new script and let me know.

      pipposan

      August 4, 2010 at 08:59

  5. Is there a way to use the CIFS repository for all the steps?
    I dont have a NFS repository.

    Marcos

    August 3, 2010 at 21:54

    • what do you mean exactly? for the backup purpose you can save the xva where you like, cifs ntfs etc etc.

      pipposan

      August 4, 2010 at 09:01

      • Let me try to explain …
        I´m not a Linux Guy.
        My config file is configured like that:
        #Directory to backup to, this should be nfs share or something large. (add / at the end)
        $backupdir = “/backup/”;

        #MountCommand is the command used to mount the directory
        $MountCommand = “mount -t cifs //172.20.251.9/backup “.$backupdir.” -o username:myuser,password=mypassword”;

        I discovered that when I ignore the option to check the free space, the script starts, take the snapshot and create a folder with the VM Name in /backup on my XenServer instead of create the folders and the .xvatemp e .xva file in my CIFS share.

        I dont know what I doing wrong or what I need to modify to solve this problem.

        Thanks for reply.

        Marcos.

        Marcos

        August 4, 2010 at 14:43

  6. I found what was wrong, username: instead username=.

    Thanks for reply and for your work on this script.

    Marcos

    August 4, 2010 at 15:57

  7. Ciao, or better is to say hi!

    Thank you for this script, I’ll test it.

    But my thank you is for the answer I’ve found in your post. I was searching how to send mails to root.

    For the moment I thank you but I not yet checked how it does work.

    Anyway, any tip about using the mail are welcome. In few words, I was (and I am) wondering how to setup the various agents (i.e. the raid control agent that should notify any issue with disks or the controller).

    So thank you if you clarify me with some tips

    Kind regards
    Robert

    Robert

    August 5, 2010 at 01:11

    • Ciao is ok as i’m italian 🙂
      btw sorry for the late reply i’m on holiday.
      are you talkin about nagios for the agent?

      pipposan

      August 28, 2010 at 11:13

  8. About the script,
    I’m reading it.
    I understand it first attempt to save snapshots.

    Many people out there says that snapshots, if used/started are really poor in performances compared with the real virtual machine.

    What about having the script that forces the use of vm-export only?

    Thank you
    Robert

    Robert

    August 5, 2010 at 01:18

    • u can do that by set the variable usesnapshot to false, btw in this case machine will be shutdown and started after export is finished.

      pipposan

      August 28, 2010 at 11:15

  9. Question.. Having issues mounting a CIFS

    Error I get

    Executing: mount -t cifs //10.1.2.24 /TEST_BLADE/ -o username=%user%,password=password

    Backup Error No space avaiable for the backup

    Executing: umount /TEST_BLADE/

    I have turned off space available.. didn’t fix it
    My windows share is wide open “everyone”.. no luck
    I have tried every combination I can think of

    FYI.. I can mount the CIFS just fine through XENCenter.. Email is working fine.. have no idea now

    BBoening

    October 14, 2010 at 22:30

    • Hi, strange that if you turn off the space avaiable the script keep lookin.
      Try to put a higher value in $spacerequired = 1000; and try to run this command:
      df -B M |grep /your/backupdirectory

      let me know so i can fix the script.
      bye

      pipposan

      October 15, 2010 at 09:15

      • What I did to get around it, was to take out the veriable..

        “mount -t cifs //10.1.2.24/backup -o username=user,password=password /backup”;

        No matter what size or turned off the “check” it would still fail.. The problem was with the mounting

        I also have other questions..
        How good are these .xva’s for backups?
        What about the meta data?
        How important is the backups of the Xen Host? backup script?

        bboening

        October 15, 2010 at 18:16

      • I will check the script to fix the problem.
        Xva generated are good and ready to import in another machine, xva generated includes metadata.

        pipposan

        October 18, 2010 at 09:03

      • I had the same problem and what worked perfectly for me was:

        Create a new ISO Library SR, pointing to your CIFS share. Name it something smart like “VM Backup Share”. After that, check with “mount” the path where it is mounted by XenServer. Use that path as your $backupdir in config.txt

        Also set $Automount to false.

        You are done. XenServer handles the mounting of your CIFS SR flawlessly, and your backup script runs smoothly.

        Gonzalo

        March 2, 2011 at 17:27

  10. Great Stuff! Quick question, I have 1 pool with 2 xenservers. When I run ‘xe sr-list’, I found 2 removeable storage (1 for each xenserver). I am guessin g I need to put two UUID for these removeable storage into $removableuuid.
    Shoudl I put something like: “55a0e4b0-629d-5fc3-193e-59e7e528837f,312fe5f5-eb0b-1715-7d90-d0af297b67fa” or put something like (’55a0e4b0-629d-5fc3-193e-59e7e528837f’,’312fe5f5-eb0b-1715-7d90-d0af297b67fa’)

    cheney

    November 24, 2010 at 08:05

    • Hi, i think for now only 1 removable storage repository is avaiable, when i have some time i will fix the script, so multiple removable storage repository will be supported.

      pipposan

      November 25, 2010 at 15:27

      • Thanks Pipposan.

        Do you know any way I can remote the removable storage from the server? If I can remote it from ‘xe sr-list’, then I should be able to run the script.

        cheney

        November 26, 2010 at 00:27

  11. Hi. Nice script, I may use it. Thanks for continuing the work of others and adding your great elements.

    One request, can you just check to see if a vm is a control node instead of asking the administrator to specify the uuid?

    It’s quite easy to do this considering the name-label for control vms will always start with ‘Control domain on host:’ as well as the dom-id will always be 0.

    So you see it’s quite easy to get this information.

    Nathan

    November 30, 2010 at 03:11

  12. Hi, forgot to mention that there is also ‘is-control-domain’ which will be true or false. I *believe* this feature was added in 5.0 and so may not be useful for a portable method, assuming you want to support 4.0, 4.1

    Nathan

    November 30, 2010 at 03:13

    • Sure when i have time i will add more options, and i’m thinkin about create a web interface for the script. But i’m a little bit busy this week.
      Cheers.

      pipposan

      November 30, 2010 at 14:52

  13. Hii! 🙂
    But as you restore the backup?
    Thanks!

    Francesco

    December 7, 2010 at 14:46

    • Simple from xencenter usually u do “import vm”.

      pipposan

      December 7, 2010 at 22:17

  14. Does this script work with 5.6 FP1?

    Tks

    Marcos Silva

    Marcos Silva

    January 28, 2011 at 16:08

  15. The script was doing great with 5.6, about 15 minutes per vc to cifs repository, now is taking about one hour ore more. The Storage I/O increased hundred times and I cant figure out what is happening.
    I did not change anything, is the same script that I was using in 5.6.

    Marcos Silva

    January 31, 2011 at 18:47

    • strange… can u post some line of /var/log/mylog or whatever name you used.
      and if the machine is big maybe the script can take some time.

      pipposan

      January 31, 2011 at 19:13

      • With XenServer 5.6

        Beginning backup of virtual machines at Mon Jan 17 01:01:03 BRST 2011

        ################################

        # Beginning backup of SPCTXES01

        ################################

        Beginning backup of 674257df-fe1d-7122-9076-85f047d2569e of the vm: SPCTXES01 @ Mon Jan 17 01:01:03 BRST 2011

        Taking a snapshot of: 674257df-fe1d-7122-9076-85f047d2569e of the vm: SPCTXES01

        Snapshot: 1d1be5db-37ed-ada3-f53d-c99f12de1987 created.

        Turning 1d1be5db-37ed-ada3-f53d-c99f12de1987 snapshot into a vm

        Exporting: 1d1be5db-37ed-ada3-f53d-c99f12de1987

        Deleting file /cifs/SPCTXES01/SPCTXES01-101220-0101.xva

        Name: /cifs/SPCTXES01/SPCTXES01-110117-0101.xvatmp – /cifs/SPCTXES01/SPCTXES01-110117-0101.xva

        Export succeeded

        Renaming backup file

        `/cifs/SPCTXES01/SPCTXES01-110117-0101.xvatmp’ -> `/cifs/SPCTXES01/SPCTXES01-110117-0101.xva’

        Done, Removing snapshot: 1d1be5db-37ed-ada3-f53d-c99f12de1987

        The following items are about to be destroyed VM : 1d1be5db-37ed-ada3-f53d-c99f12de1987 (SPCTXES01-backup_vm)
        VDI: 16f48b42-5ff6-4f8c-8f47-29f2683c41a6 (0) All objects destroyed

        Completed backup of of the vm: SPCTXES01 elapsed minutes: 11.6833333333333

        —————————————————-
        With XenServer 5.6 FP1

        Beginning backup of virtual machines at Sun Jan 30 17:56:54 BRST 2011

        ################################

        # Beginning backup of SPCTXES01

        ################################

        Beginning backup of 674257df-fe1d-7122-9076-85f047d2569e of the vm: SPCTXES01 @ Sun Jan 30 17:56:54 BRST 2011

        Taking a snapshot of: 674257df-fe1d-7122-9076-85f047d2569e of the vm: SPCTXES01

        Snapshot: 434e7e68-0d60-1d2c-ec3f-66752b6a8ae8 created.

        Turning 434e7e68-0d60-1d2c-ec3f-66752b6a8ae8 snapshot into a vm

        Exporting: 434e7e68-0d60-1d2c-ec3f-66752b6a8ae8

        Deleting file /cifs/SPCTXES01/SPCTXES01-110103-0101.xva

        Name: /cifs/SPCTXES01/SPCTXES01-110130-1756.xvatmp – /cifs/SPCTXES01/SPCTXES01-110130-1756.xva

        Export succeeded

        Renaming backup file

        `/cifs/SPCTXES01/SPCTXES01-110130-1756.xvatmp’ -> `/cifs/SPCTXES01/SPCTXES01-110130-1756.xva’

        Done, Removing snapshot: 434e7e68-0d60-1d2c-ec3f-66752b6a8ae8

        The following items are about to be destroyed VM : 434e7e68-0d60-1d2c-ec3f-66752b6a8ae8 (SPCTXES01-backup_vm)
        VDI: 53476ea9-54ab-424d-a58f-0a5ce5ad06f8 () All objects destroyed

        Completed backup of of the vm: SPCTXES01 elapsed minutes: 104.85

        Completed backup of 674257df-fe1d-7122-9076-85f047d2569e of the vm: SPCTXES01 @ Sun Jan 30 19:41:45 BRST 2011

        Marcos Silva

        January 31, 2011 at 21:58

      • seems all is ok, what is the size of the machine?

        pipposan

        February 3, 2011 at 13:28

  16. Pipposan,

    The strange thing is that export size increased a lot.
    Before the upgrade the export size was:

    VM01 – Disk Size 20G – Used 12G – Export Size 13G
    VM02 – Disk Size 40G – Used 11G – Export Size 11G
    VM03 – Disk Size 40G – Used 13G – Export Size 13G

    After the upgrade:

    VM01 – Disk Size 20G – Used 12G – Export Size 33G
    VM02 – Disk Size 40G – Used 11G – Export Size 20G
    VM03 – Disk Size 40G – Used 13G – Export Size 13G

    I dont know why the export size increased and why the storage IOPS reached millions values.

    Before the upgrade to FP1 the IPOS during the backup was between 3/5K IOPS.
    Now during the backup the IPOS is 2 millions, when this happen my entire environment freezes because the controller cand address until 20K IPOS.
    You can see this strange behavior in this picture http://i53.tinypic.com/2l9ktv8.jpg.

    Regards,

    Marcos Silva

    Marcos Silva

    February 4, 2011 at 15:52

    • Strange i think it’s related with FP1 try to open a bug or i see you post on citrix forum, let’s wait a response, in meantime i will do some test, if i found something i report you back.
      cheers

      pipposan

      February 9, 2011 at 11:19

  17. Wonderful script! Really good work.
    Runs out of the box on xs 5.6 here, no problems at all.

    JoeL

    February 12, 2011 at 13:18

  18. Works fine!!! Thanks for that.

    What about a feature for make a zip file out of the machine? It saves a lot of storage. Is it possible?

    Cheers

    Timm

    March 3, 2011 at 12:57

  19. I’ve put this script (2.1) in production (xs 5.6.0fp1). It works great, except that it doesn’t delete the VDIs of the snapshots with the vm-uninstall.

    If I uninstall a vm from Xencenter it works great, but not from CLI. Anyone had the same experience? It is weird to finish with hundreds of unassigned VDIs in the SR.

    Eduardo Diaz

    March 23, 2011 at 20:24

    • if you look in the log or in the mail, the script try to uninstall the snapshot?

      pipposan

      March 25, 2011 at 10:05

      • Yes… As an example:

        Thu Apr 7 04:32:51 2011 The following items are about to be destroyed
        VM : 32a7dd90-fde3-2043-f1df-efdb7b4afdd0 (Appwebserver-backup_vm)
        All objects destroyed

        But in fact, the snapshot is deleted and the drives are left behind.

        It is not critical, but is not funny to find a lot of vdis in the storage reopository 😦

        Eduardo Diaz

        April 8, 2011 at 00:22

      • it’s a windows machine?
        i found this:
        http://forums.citrix.com/message.jspa?messageID=1545769

        pipposan

        April 8, 2011 at 10:01

      • YES! It is a windows based VDI. This is the data I can find about one of those “undeleted” VDIs…

        Maybe the “managed: true” has something to do with it. Also the “is-a-snapshot” may be relevant. Any clue?

        # xe vdi-list params=all uuid=22c34bbe-4cb0-42c3-a6f5-aeece3793044
        uuid ( RO) : 22c34bbe-4cb0-42c3-a6f5-aeece3793044
        name-label ( RW): temperaturas_0
        name-description ( RW):
        is-a-snapshot ( RO): true
        snapshot-of ( RO): 2642d80d-4c26-475a-9dab-c8968d0f0386
        snapshots ( RO):
        snapshot-time ( RO): 20110408T01:45:19Z
        allowed-operations (SRO): resize; destroy; clone; copy; snapshot
        current-operations (SRO):
        sr-uuid ( RO): da0fd7dc-0ac7-45ba-88b4-f164acf4e6f8
        sr-name-label ( RO): Local storage
        vbd-uuids (SRO):
        crashdump-uuids (SRO):
        virtual-size ( RO): 7516192768
        physical-utilisation ( RO): 8388608
        location ( RO): 22c34bbe-4cb0-42c3-a6f5-aeece3793044
        type ( RO): User
        sharable ( RO): false
        read-only ( RO): false
        storage-lock ( RO): false
        managed ( RO): true
        parent ( RO):
        missing ( RO): false
        other-config (MRW):
        xenstore-data (MRO): vdi-type: vhd; storage-type: lvm; scsi/0x12/0x83: AIMAMQIBAC1YRU5TUkMgIDIyYzM0YmJlLTRjYjAtNDJjMy1hNmY1LWFlZWNlMzc5MzA0NCA=; scsi/0x12/0x80: AIAAEjIyYzM0YmJlLTRjYjAtNDIgIA==; vdi-uuid: 22c34bbe-4cb0-42c3-a6f5-aeece3793044
        sm-config (MRO): vhd-parent: 707c096b-40ce-4ddf-95fd-d8f0e9d05c70; import_task: OpaqueRef:2a36b527-ad04-8173-a9a4-fda1beda5784; vdi_type: vhd
        on-boot ( RW): persist
        allow-caching ( RW): false
        tags (SRW):

        Eduardo Diaz

        April 8, 2011 at 11:10

  20. I use the script on a 5.6 xen server but i can not import the backup files

    rexolb

    March 24, 2011 at 22:10

    • what do you mean you can’t import the backup files?
      an error occured? you don’t know how to import?
      please be more verbose 🙂

      pipposan

      March 25, 2011 at 10:07

      • Sorry for my short description

        I’m using xen server 5.6 fp1
        i backuped a 2008 server using your script 2.1
        When using xencenter to import the device he start the inmport after 40 sec he gives the error can not import this device

        Rexolb

        March 25, 2011 at 11:07

      • strange only this error is diplayed on xencenter?

        pipposan

        April 7, 2011 at 18:09

  21. i get this error how can i fix it?

    Operation failed. Error: No such file or directory: open, /backup/JANSENTD/JANSENTD-110406-1415.xvatmp.
    mv: cannot stat `/backup/JANSENTD/JANSENTD-110406-1415.xvatmp’: No such file or directory

    Eldar

    April 6, 2011 at 14:17

  22. After successful backup windows 2003 r2 sp2 (AD) my vm get high cpu utilization with wmiprvse.exe…. any body have this bug?

    Gerasim

    July 14, 2011 at 12:40

  23. Hi,

    I have a question about this script but first of all, this script is the best that I have seen.
    But, I’m sorry, I have one question,
    After a backup run, some vm disk are still available on my storage. I delete this files (snapshots) manualy but some other files are deleted by this script…..
    Is there something that I doing wrong??

    thanks for help!

    regards,
    Dirk

    Dirk

    October 4, 2011 at 22:56

  24. Hi Pipposan,

    I am relatively new to XenServer administration and was just curious as to whether or not you’d be able to assist in a problem I am having.

    What I am trying to accomplish here is to backup my VM’s to an External HDD (Connected directly via USB to the XenServer Host)

    I have downloaded your script and placed it in the root directory of my XenServer.

    I have made the changes in Config.txt and Added Dom0 to the @skip variable.
    I ensured the correct UUID was put in the @removableuuid variable – I retrieved the UUID via xe sr-list

    When I attempt to run the script, it tells me that there is insufficient disk space to execute the backup (Even though the external is formatted as EXT3 1000.0GB)

    I could possibly be missing a few steps here, so a rough guide on what steps to take to make this function for me would be greatly appreciated.

    Regards,
    WebJunkie2.0

    Vinny

    November 17, 2011 at 08:28

  25. Hi. i have error occured.
    Backup Error No space avaiable for the backup at backup.pl line 83.
    i`ve using local storage for backups, and i`ve mounting it across fstab. how i can fix this error?

    The_KOPACb

    December 19, 2011 at 12:01


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: