If you also have some interesting stories to tell about the pandemic, write them down and we will publish them! Tell us about your solutions in teaching, experimenting and research! We are curious how you managed the last year! Send your stories to Melanie.Nentwich@desy.de
A beamtime at a synchrotron is always a lot of stress and less sleep. However, during a pandemic, completely new challenges were waiting for us.
We are Tina from the Institute of Experimental Physics, TU Bergakademie Freiberg, and Melanie from beamline P23 at PETRA III, DESY. We were colleagues for many years and had many experiments, conferences and office time together. Now, that Melanie is a PostDoc at their favourite beamline, of course some things have changed. However, due to the pandemic, more things changed than expected and hoped. Here, we want to give you a short comparison of our latest experience including different views on the situation.
(As the proposal was as always, we skip this part and perhaps describe it in detail in a different blog post as a HowTo.)
Preparation
When the beamtime approaches, the samples need to be prepared. The single crystals need to be cleaned, pre-oriented (to know surface and edge planes, which simplifies the later orientation at the beamline) and coated with electrodes.
Additionally, our sample chamber and its periphery needs to be repaired, complemented and finally packed up.
The day before beamtime starts, we get the rental car and load the boxes for our 4h to 7h ride from Freiberg to Hamburg. Step the gas, Tina!
This time, everything was different. Melanie now was working for DESY and could not help during the preparations. Additionally, in the very last moment, Tina and her colleagues were not allowed on the business trip anymore, due to the pandemic. Luckily, the beamline manager was very flexible and made it a mail-in-beamtime.
While the packages were sent with overnight express to Hamburg, Melanie prepared herself to be responsible for EVERYTHING happening. As this was only her second beamtime as local contact, this was a scary idea!
Beamline Control
For our experiments, we want to measure a Bragg peak while changing the energy.
So, first, we need to find two arbitrary Bragg reflections to orient the crystal relative to the machine, which is a lot easier, when the crystal was pre-oriented in the lab. Then it is easy to find the reflection that is important for the experiment. Afterwards, we move (virtual) energy motors. Simultaneously, we need to keep the Bragg condition, which means moving all diffractometer circles at once and sometimes to very unusual positions.
The aim of our experiment is to measure the energy dependent intensity of reflections as they can be sensitive to structural changes. In our in-situ experiments, we determined pico-metre atomic displacements (read more here)!
We control the motors with the spock software listening to the buzzing of the motors next door. We can check the movement of the goniometer with two cameras, which enable us to see almost the whole experimental hutch. Additional software is necessary, e.g. to start spock or to visualize the detector images; the corresponding interfaces are scattered on 6 huge monitors.
Our main goal is to combine many commands into longer scripts so that we get a break. For overnight measurements, it is extremely important to consider all eventualities, like beam loss, reflections sliding out of the scanning range… Even with our scripting expert Carsten, we are mostly only ready at 3 am.
In theory, nothing changed too much as the off-site colleagues used remote control for the main computer and the motors. But the devil is in the details.
Conducting a beamtime from home sounds comfy. Laying on your sofa in your pyjamas and snacking in the meantime. However, all the colleagues from your institute know that you are home and think you had time for other things besides your beamtime.
As Tina and the other two colleagues working from home only had one small laptop screen and maybe a larger external screen, they had to open a bunch of tunnels to the control computers at DESY and handle many windows for remote control of the beamline software and cameras in the hutch. There were always too few windows and always one of the colleagues at home had some internet connection problems (mostly Tina).
At some point during the remote beamtime we recognized that losing or cutting the connection to the control computer in combination with some software problems, caused the control software to crash. Somehow this always happened during the night scan: the most productive time to accumulate data and the only chance to sleep.
During the first night scan the software crashed right after we all went to bed and disconnected the remote control. Tina spent the whole next night in front of her PC and watched if everything was working and nothing crashed until 6 am when she went to bed. One hour later, when the early shift aka Melanie came, the software had already crashed. At least we didn’t lose the complete night. For the last two nights, Carsten wrote a python script, which checked if the detector was still making frames. If not, an alarm starts. So we could get sleep and we only needed to get up (woken by a shrill siren) to restart the measurement
Discussions
Of course, normally we discuss directly in person. In the experimental hutch or in the control hutch, wherever the problem occurs. All members bring their very specific expertise to the experiment: sample preparation in general, material details, scripting, hands-on experience with synchrotron radiation, 3D visualization of the diffractometer movement… This combination is the strength of our team and already prevents us from many mistakes.
Communication became even more important for the mail-in beamtime! Because you couldn’t always see what your colleagues did, it was important to speak about what you do or want to do. The spock terminal did not show what other people were typing, only the results of their action. If all decided to move a motor, we had to make sure that exactly one person entered the commands.
We used the video conference system Big Blue Button to connect all the beamtime coworkers with each other. Of course, Melanie could not really carry it to the experimental hutch. Luckily, this conference system allowed us to lock in via normal telephone calls. To share the visual details of sample position and other things with tons of photos via messengers.
Melanie often seemed alone in the experimental hutch, but the comrades far away were always spying on every movement using the cameras in the experimental hutch
During the time Melanie was working, the colleagues at home had a short break. This meant time to grab some food or do all these things that you usually do at home (cooking, cleaning dishes, starting a laundry,..) or to explore the features of BBB. When they recognized it was the third advent, they started to share their favorite christmas songs via BBB (upload a youtube link and BBB play the song synchronously for all participants) and had a little christmas party
Conducting the Experiment
Tina is responsible for putting the samples on our sample holder, as she has the most experience.
We are all working together to put the equipment into the experimental hutch, fixing screws, pipes and hoses until all the supply is unpacked and attached. This teamwork makes it easier to see if something is wrong in the experimental set-up. If we need to adjust, optimize or change the set-up during the experiment, we are always helping each other and discussing the improvements to continue fast with the measurement
This time, all the handy work was left to Melanie, the Applied Mathematician, dealing with theoretical crystallography and doing the sample preparation for the first time. Sounds like a bad idea?
Well, it was not. Melanie showed a good hand in preparing the samples. She gave her very best by attaching the equipment. (Yes, she did!) Unfortunately, some minor parts were forgotten while packing. So a DESY-wide search started for a Triax-to-Coax-Adapter. Did you know those things are rare? Well, Tina and Melanie know now.
Melanie learned a huge lot during a very small time. Exhausting but fulfilling, when things are working.
Normally, we need only one sample per beamtime for this kind of experiment. Unlikely, we had this time some troubles with the samples and Melanie needed to mount and conduct the samples FOUR times. All just about to train Melanie’s sample preparation skills ;)
She needed to control the set-up all the time on her own. Luckly, everything was perfect. And this means, it was also the perfect time to go home for her. Unfortunately, a few hours later we needed to change the set-up or killed the sample. ”Melanie, could you come back to DESY, please?”
“Maintaining” Ourselves
How to stay motivated and in a good mood while giving your best during a stressful beamtime? At least have good food. Luckily, the food deliveries may enter the DESY campus and bring your hot, well-earned food directly to your beamline door at 10 pm ;)
Here are our recommendations:
- Sweet Mint
- MostWantedBurger (Eimsbüttel)
- Osaka Sushi
- Pizza
- repeat ;)
And not to forget: long breakfast on Sunday at Bäckerei Junge, Osdorfer Weg 106.
Before the beamtime Tina cooked a big pot of pumpkin soup to have a warm meal and don’t waste so much time cooking. At some point she recognized that she didn’t leave her apartment for three days. Stay at home, fight the virus ;)
Cleaning up
Beamtime ends for us normally at a machine day (for service) at 7 am, with a lot of time to clean the beamline up. The next users will only arrive the next day. Enough time to get some sleep.
The colleagues at home had nothing to do on this day and could rest, but it felt very bad to leave all the work to one.
The colleagues at home had nothing to do on this day and could rest, but it felt very bad to leave all the work to one.
Unfortunately, really everything was different than usual. No machine day after us, but the next users want to start at 9 am.
Melanie had to get up very early and started cleaning up at 7 am. After two hours, the boxes were neatly packed and all screws, vacuum clips, and screw drivers were back at their places. Time to get home and sleep…
Summary
It was a great experience to have a remote beamtime. Although almost everything was different, it felt like a beamtime as the level of stress, sleep and daylight was almost the same as for a regular beamtime ;) We can really imagine that controlling the beamline from home will be the future. Remote beamtimes could reduce the traveling of scientists over the whole world and are therefore saving costs and time, and they could also bring an ecological advantage as well ;)
Joining a beamtime from home would be less stressful in terms of visa, flight connections, and accommodation etc. especially for people with family and/or great responsibility at their home institute. Additionally, more people could join the beamtime than the three people that are usually reimbursed by the synchrotron facility. This could be a good opportunity for undergrad and fresh PhD students to gain experience with synchrotron experiments, but also for cooperation partners to easily join the beamtime exactly when their experience is needed. But, of course one or two people should be on-site, to quickly react to any troubles that could happen during your experiment and to reduce the stress for the beamline scientists. Because no one knows your samples and experiments better than you ;)
Finally, we want to thank the flexible and helpful beamline staff at P23! Thank you so much for providing the remote control and for immediate help every now and then with improving scripts and creating workarounds! Without you, our beamtime would not have been such a success!