Medical IT System Support
Recovering millions of documents in a database.
Every IT product is intended to meet certain business needs, e.g., generate medical documents (such as certificates or prescriptions) automatically and provide convenience to doctors and patients.
To ensure the IT system performs these tasks, it is critical to arrange workflows for both development and technical support. In this case study, we explain how we helped to recover a healthcare IT system and increase its safety.
Where did we begin?
Our partner approached us to test and enhance the functionality of an IT system employed by European clinics for doc-flow purposes. Doctors and administrators used the program to process thousands of files and appointment records on a daily basis, while millions of files were stored in a database.
In this project, we performed a lot of tasks:
– The client needed to expedite new releases in a short time frame, so they involved us in product development and testing.
– We also supported their "cloud," with some clinics installing a desktop program on their servers independently.
System incident
After a year of successful cooperation, the client requested that we perform a new task. There was a database failure at one of their clinics. This happened just before Christmas, so the client did not have enough time for recovery, since Europe was about to go on a long holiday. The problem was that the appointment records of more than a million patients had been damaged. The physical disk containing a virtual machine was also damaged, and the clinic did not maintain backup copies. The only option we could arrive at was to recover the data from a hard disk backup.
When we analyzed the system settings, we encountered another problem. We found out that restoring data from a backup copy, as we initially intended, was impossible. The disk backups were not saved, since the clinic only allocated 500 Gb for them on the server, which was insufficient.
Looking for alternative options, we detected a curious error. It turned out that part of the database information was copied to ElasticSearch, although this is usually unnecessary. It was exactly this error that bailed us out.
To recover the files, we wrote a utility that combined the database and ElasticSearch content. We tested it and made sure it worked. Despite partial database damage, we employed a few different approaches and succeeded in saving all the data. Over the Christmas holidays, we recovered all the lost records — more than 2 million documents. The client was happy with the result and thanked us.
Conclusion
After "rescuing" the system, we offered to help with its thorough inspection and setup. First, we checked the servers in other clinics in addition to the faulty server. We put together our safety recommendations based on the inspection results, obtained the client's approval, and implemented them.
We set up backup with SQL Server and provided additional monitoring of backups on our side and a night-time alert system. We are currently keeping an eye on the infrastructure of all clinics that have the IT system installed. Due to the timely help from our team, the client successfully recovered their IT system and prevented possible risks in a short time frame. This project is yet another example demonstrating how we help our clients in all IT product stages, from creation to maintenance and support.