Abstract
This letter describes an electromagnetic feedback control system for rigid-body motion control of a magnet. Its novel features are that sensing and actuation using magnetometer sensors and actuator coils operate simultaneously, and magnetic field models from the controlled magnet and each of the actuator coil currents are used together to calculate the 3D position and orientation of the magnet to control motion simultaneously and independently in multiple degrees of freedom including planar translation and two in rotation, leaving rotation about the cylindrical axis of magnetization uncontrolled. The system configuration and the localization and actuation methods are presented with experimental results of magnet localization with constant and varying coil currents, and during feedback control of trajectory following motion of the magnet in multiple directions on a planar surface and with controlled changes in orientation. The intended application of the system is for motion control of magnetic endoscope capsules and other miniature medical devices inside the human body.
I. Introduction
Electromagnetic motion control of medical devices inside the human body is an active topic of research and development as a minimally invasive means to access internal anatomy for diagnostic purposes such as endoscopic capsules, and potentially for therapeutic purposes such as excising tumors or targeted drug delivery. We have developed a system which uses planar arrays of magnetometer sensors and actuator coils for feedback control of the motion of a cylindrical magnet using electromagnetic sensing and actuation simultaneously. The novelty of our system is that magnet orientation and position can each be controlled independently in multiple directions without using any additional hardware such as optical sensing, external magnets, manipulator arms, or mechanical actuation components, and the magnet motion does not depend on its shape or contact with its environment in any way.
Other research projects [Tognarelli et al., 2012, Keller et al., 2012, Quaglia et al., 2009] have combined magnetic propulsion with magnetic or other position sensing for endoscope capsules, but with limited motion control directions or ranges. In [Chiba et al., 2007], a helical ridge on the capsule is used to produce linear motion from a rotating field, so that propulsion is generated by combined electromagnetic and mechanical means. The OctoMag [Kummer et al., 2010] system uses multiple electromagnets with optical sensing to control microbot position and orientation in a small volume for intraocular procedures. Other examples of magnetic capsule propulsion with additional hardware are in [Honda et al., 1996, Ishiyama et al., 2001, Chiba et al., 2005, Yesin et al., 2006, Sudo et al., 2006, Sendoh et al., 2003, Carpi and Pappone, 2009, Swain et al., 2010] with in vivo and feasibility trials in [Morita et al., 2010, Rey et al., 2010, 2011, Rupprecht et al., 2012]. Other systems with large external magnets for force generation are described in [Natali et al., 2013, 2016], with control of a untethered single magnet in fluid with optical tracking in [Mahoney and Abbott, 2016], and through the gastrointestinal tract in [Ciuti et al., 2010].
Several research groups have combined localization from an array of Hall-effect magnetometer sensors with a rotating magnetic field to produce magnet capsule motion with coupled rotation and translation by rolling without slip on a flat surface [Son et al., 2016] or by helical motion to swim through a fluid in a narrow tube [Popek et al., 2017b,a, Khalil et al., 2019]. These motions are limited to a single direction, with coupled rotation and translation, and depend on the magnet capsule shape and contact state with its environment to produce locomotion. A bulky external magnet is used to produce the rotating magnetic field. Our system and methods described in this letter are not subject to any of these limitations, as the compact 10-coil actuation array can directly produce forces and torques on the magnet independently in any desired direction excepting torques about the magnetization axis, and without regard to the capsule shape or contact state.
Capsule rotation and translation are both controlled in [Son et al., 2020], but propulsion is generated by a rotating field and rolling contact, and forces can only be generated along the magnetization axis. In [Pittiglio et al., 2019, Taddese et al., 2018] magnet orientation and translation are controlled together, but a robot arm and permanent magnet must be used above the magnet, and the tethered capsule includes Hall effect sensors and an inertial measurement unit.
Multiple coils acting together can generate fully three-dimensional force and torque vectors in any direction on a rigid body containing two or more magnets, or five degree of freedom actuation on a single cylindrical magnet. If a motion tracking system is used to sense position and orientation of the magnets in real time, then feedback control may be implemented to achieve stable magnetic levitation [Berkelman and Dzadovsky, 2010]. We have also used flat rectangular coils with an array of single-axis magnetometer sensors [Liang et al., 2012] for 2D [Berkelman et al., 2018] and 3D [Berkelman and Abdul-Ghani, 2020] force generation for haptic interaction.
The following sections describe the basic method for magnet localization, its modifications for magnet localization while multiple coil currents are applied, implementation details, localization and feedback control results, and a discussion and conclusion.
II. Magnet Localization
A single cylindrical magnet may be localized in both position and orientation using an array of magnetometer sensors, by finding the three components of the magnet center position P and two components of the unit vector orientation of its magnetic axis H0 which minimize the differences between the field measurements obtained by the sensors and those predicted by the magnetic dipole model:
(1) |
where Blx, Bly, and Blz are the three components of the vector field Bl, with μr the relative permeability of the medium, μ0 the magnetic permeability of air, and MT as a constant depending on the overall magnet strength. H0 is the unit vector describing the orientation of the axis of magnetization, P is the vector from the magnet center to the sensor, and R is the distance to the magnet center.
For an array of sensors at known locations Pi the magnetic dipole equation is
(2) |
where the constant BT combines the magnetic permeability and magnet strength constants from (1) and is determined experimentally for a given magnet.
The magnetic dipole model of equations (1) and (2) models the magnetic field as generated from a single point and should therefore be used only in cases where the sensor to magnet distance is at least an order of magnitude greater than the dimensions of the magnet.
The Levenberg-Marquardt (LM) nonlinear solution method [More, 1978] is used to find magnet position and magnetization axis orientation unit vector H0. This localization method is most similar to those described by [Hu et al., 2006, 2005], in which a 4×4 array of three-axis magnetometer sensors has typically been found to be sufficient for localization to an accuracy of 2 mm or better.
III. Magnet Localization with Coil Actuation
For magnet localization while coil currents are active, the expected magnetic fields produced by each coil individually are subtracted from the sensor signals before applying the LM solution method of the previous section. Due to the absence of ferrous materials and variable magnetization effects in the system it is assumed that magnetic fields from the coils and magnet are independent and may be superposed as
(3) |
where Bt is the total magnetic flux density field, Bm is the field surrounding the magnet to be localized, and Bi are the fields produced by each cylindrical coil individually, summed over n coils.
The models of the field produced by each coil individually at each sensor location are produced by direct measurements and not the dipole model. Direct measurement of individual fields is preferable in this situation due to potentially significant unmodelled variations in the dimensions, positions, and generated currents in each coil, and because the distance between the center of each coil and the localized magnet may be less than an order of magnitude greater than the sizes of the coils and the magnet.
Other applications with simultaneous magnet position detection and coil actuation for haptic interaction are described in [Berkelman et al., 2018] and [Berkelman and Abdul-Ghani, 2020], however flat coils were used and the localization was limited to the planar position of the magnet. In that situation the fields generated by the coil were significantly less than the fields in the vicinity of the magnet, and the magnet could be localized by discarding the sensor values less than a given threshold and finding the centroid of the remaining field values in the vicinity of the magnet. A similar system demonstrated magnet localization in the vicinity of cylindrical coil currents by using sensor arrays both above and below the coil arrays so that the difference in the signals from the top and bottom arrays corresponds to the field of the magnet by itself [Adel et al., 2019].
IV. Hardware and Software Design
A 4×4 array of 16 Honeywell HMC5983 digital compass magnetoresistive sensors, supplied on breakout boards from Drotek Electronics, was used for electromagnetic sensing. Each sensor produces 12-bit measurements of magnetic flux density in three axes, within a measurement range of ±0.8 mT. The sensors were fixed to a circuit board with a spacing of 38.0 mm between neighboring sensor centers in both directions. The sensors communicate with an Arduino Due microcontroller through a Serial Peripheral Interface (SPI) bus, and the Arduino board communicates through a native USB port with a Linux host computer which performs the localization and feedback control calculations.
An array of 10 cylindrical coils, closely packed together in rows of 3, 4, and 3 coils, was used for electromagnetic force actuation. Each coil is 25 mm in diameter and 30 mm in height, with 1000 windings for a resistance of approximately 8 Ω. Current amplifiers (4212Z, Copley Controls Inc.) configured for the 3.5 mH impedance of the coils are controlled by the host PC. The coil array can generate forces and torques in any direction on magnets up to 40 mm above it by activating coil currents in combination according to a coil current to force and torque vector transformation, which is continuously updated as the magnet position changes, so that its pseudoinverse can be used to calculate the least-squares array of coil currents needed to generate the desired force and torque vector for feedback control, as described in [Berkelman and Dzadovsky, 2013]. The design and analysis methods for the coil array actuation are described in [Berkelman and Dzadovsky, 2010].
The system including actuation coils, sensors, and the controlled magnet is shown in Fig. 1 with the sensor array moved forward from directly underneath the coils for visibility. The coil and sensor arrays are vertically separated so that the combined magnetic fields produced by the magnet and coil currents up to 1.0 A do not exceed the measurement ranges of the sensors. The system was validated with a cylindrical NdFeB magnet, 6.35 mm thick and 19.05 mm in diameter, and grade N52 supplied from K & J Magnetics and magnetized through its axis of rotation.
Fig. 1.
Electromagnetic Motion Control System
The system is initialized by first reading the value of each sensor without any nearby magnet or active coil currents, to obtain a baseline value for each sensor channel corresponding to sensor offsets and background fields. Next, each coil is actuated individually in turn at the 0.5, 1.0, −0.5 and −1.0 A levels and the average ratio of coil current to the change in field is calculated between each of the 10 coils and 48 sensor signals so that the Bi values from (3) can be calculated. Last, the magnet localization procedure is executed with the magnet fixed to the coil array with no active currents, to obtain the offset between the coil and sensor array coordinate systems. This initialization procedure serves to calibrate the system, accounting for unmodelled variations in coil position and currents, so that precise coil field models and measurements are not needed for localization.
The C/C++ Minpack library [Devernay, 2007] implementation of the Levenberg-Marquardt solution method was used for localization using the lmdif1() function. Due to the efficiency of this computational library and the computational speed of present PCs, the primary limitation of the system update rate is the sampling rate of the sensors, not the localization computations.
Occasional errors in the current amplifier or magnetometer sensor signals during operation are detected in real time whenever the residual value obtained during the Levenberg-Marquardt localization is significantly greater than its typical value. Therefore, whenever the final Levenberg-Marquardt residual error value obtained from the localization algorithm exceeds its typical value by more than 50 percent during system operation, the localization data is discarded and not updated for feedback control.
V. Experimental Validation
The accuracy of the sensing and localization methods and implementation as described were evaluated as the magnet was moved by motorized precision motion stages (Unislide and Xslide from Velmex, Inc.) along a trajectory made up of straight line motions of 25 and 50 mm in all three principal directions. As the magnet was moved, localization was performed first with no coil currents, then with constant random currents within a range of ±1.0 A in each coil, and then with continuously updated random currents within a range of ±1.0 A in each coil, using the method described in Sections II and III. The motion trajectory and localization results are shown in Figs. 2 and 3. These results show that magnet localization can be done with this system in the presence of coil currents, although measurement noise of approximately 1 mm may be introduced in the sensed position.
Fig. 2.
Localization results in 3D during stage motion using sensor array with and without coil currents
Fig. 3.
Localization results versus time during stage motion using sensor array with and without coil currents
Motion control with simultaneous electromagnetic localization and actuation was demonstrated by controlling the magnet to follow a 10 mm square trajectory on a horizontal surface approximately 8 mm above the coil array. A proportional position error control gain of 0.03 N/mm was used to provide gentle guidance to move the magnet with minimal forces. Commanded and sensed trajectory following position results are shown in Figs. 4 and 5. Although motion irregularities can be seen in the position data due to static friction and noise in the sensor data and position localization, the motion trajectory is successfully executed.
Fig. 4.
Magnet motion control results during planar trajectory motion
Fig. 5.
Magnet motion control results versus time during trajectory motion
Trajectory motion control with independently varying rigid-body position and orientation commands is shown in Fig. 6. For these results the magnet was mounted on an inverted hemispherical base, lubricated with graphite to minimize surface friction which would constrain free independent rotation and translation of the magnet on the plane. The proportional rotational error control gain was 4.0 N-mm/rad. Position and rotational error derivative gain damping was applied for stability, sensed position and orientation were filtered by a moving average over three samples for smooth motion, and the control rate was 100 Hz. These results demonstrate the novel capability of the system of independent, simultaneous motion control with variable position and rotation command trajectories, without use of an external magnet or robot arm.
Fig. 6.
Magnet translation and rotation trajectory control results
VI. Discussion and Conclusion
This work has demonstrated that trajectory following feedback motion control of a magnet can be done in multiple directions, with independent rotation and positioning, using electromagnetic sensing for position localization simultaneously with actuation by multiple currents in cylindrical coils.
The novel capabilities of our system could enable more sophisticated tasks to be performed by an endoscopic capsule as compared to motion in limited directions and with coupling between translational and rotational motions. For example, a miniature camera, needle, or grasper must often be correctly oriented as well as positioned in order to perform its task. Similarly, independent control of orientation and position is typically necessary for mechanical part assembly. For these situations an important distinction can be made between simple locomotion in a desired direction, and our system’s capability of 3D spatial rigid-body manipulation in which position and orientation are controlled independently without using any external magnet or robot arm.
The results shown are an initial proof of feasibility, and magnet motion control accuracy and speed are expected to be able to be improved through the use of more precise sensing and current generation. Newly available magnetometer sensors such as the RM3100 Geomagnetic Sensor from PNI Corp. claim order of magnitude improvements in noise level, measurement resolution, and sample rate as compared to the sensors currently in use. Use of linear current amplifiers to drive the coil actuation rather than the pulse width modulation amplifiers used at present may also reduce the noise level in the sensed magnetic fields, improving localization accuracy. Finally, increases in the size of the actuation coils and their currents is expected to increase the forces and operating range of the system to where these are sufficient for five degree of freedom wireless rigid body motion control of an endoscopic capsule in the human body.
Acknowledgement
Support was provided by National Institute of Biomedical Imaging and Bioengineering (NIBIB) grant #R03EB018593 and the University of Hawaii College of Engineering.
Contributor Information
Peter Berkelman, Department of Mechanical Engineering at the University of Hawaii-Manoa, Honolulu, HI, 96822 USA.
Bernadette Tix, Information and Computer Sciences Department at the University of Hawaii-Manoa..
References
- Adel A, Mansour M, Micheal MM, Abdelmawla A, Khalil ISL, and Misra S. Magnetic localization for an electromagnetic-based haptic interface. IEEE Magnetics Letters, 10, March 2019. [Google Scholar]
- Berkelman P and Abdul-Ghani H. Electromagnetic haptic feedback system for use with a graphical display using flat coils and sensor array. IEEE Robotics and Automation Letters, 5(2), April 2020. [Google Scholar]
- Berkelman P and Dzadovsky M. Novel design, characterization, and control method for large motion range magnetic levitation. IEEE Magnetics Letters, 1, January 2010. [Google Scholar]
- Berkelman P and Dzadovsky M. Magnetic levitation over large translation and rotation ranges in all directions. IEEE/ASME Transactions on Mechatronics, 18(1):44–52, 2013. [Google Scholar]
- Berkelman P, Tix B, and Abdul-Ghani H. Electromagnetic position sensing and force feedback for a magnetic stylus with an interactive display. IEEE Magnetics Letters, 10:1–5, December 2018. [Google Scholar]
- Carpi F and Pappone C. Magnetic maneuvering of endoscopic capsules by means of a robotic navigation system. IEEE Transactions on Biomedical Engineering, 56(5):1482–1490, 2009. [DOI] [PubMed] [Google Scholar]
- Chiba A, Sendoh M, and Arai KI. Basic characteristics of a magnetic actuator for capsule endoscope. Transactions Japan Society of Medicine, Biology and Engineering, 42:343–346, 2005. [Google Scholar]
- Chiba A, Sendoh M, Ishiyama K, Arai KI, Kawano H, Uchiyama A, and Takizawa H. Magnetic actuator for a capsule endoscope navigation system. Journal of Magnetics, 12(2):89–92, 2007. [Google Scholar]
- Ciuti G, Valdastri P, Menciassi A, and Dario P. Robotic magnetic steering and locomotion of capsule endoscope for diagnostic and surgical endoluminal procedures. Robotica, 28(2):199–207, 2010. [Google Scholar]
- Devernay F. C/c++ minpack. http://devernay.free.fr/hacks/cminpack/, 2007.
- Honda T, Arai KI, and Ishiyama K. Micro-swimming mechanisms propelled by external magnetic field. IEEE Transactions on Magnetics, 32:5085–5087, 1996. [Google Scholar]
- Hu C, Meng MQ-H, and Mandal M. Efficient magnetic localization and orientation technique for capsule endoscopy. International Journal of Information Acquisition, 2(1):23–36, March 2005. [Google Scholar]
- Hu C, Meng MQ-H, Mandal M, and Wang X. 3-axis magnetic sensor array system for tracking magnets position and orientation. In World Congress on Intelligent Control and Automation, page 53045308, June 2006. [Google Scholar]
- Ishiyama K, Sendoh M, Yamazaki A, and Arai KI. Swimmimg micro-machine driven by magnetic torque. Sensors and Actuators, A-91:141–144, 2001. [Google Scholar]
- Keller H, Juloski A, Kawano H, Bechtold M, Kimura A, Takizawa H, and Kuth R. Method for navigation and control of a magnetically guided capsule endoscope in the human stomach. In IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics, pages 859–865, Rome, June 2012. [Google Scholar]
- Khalil ISM, Adel A, Mahdy D, Micheal MM, Mansour M,Hamdi N, and Misra S. Magnetic localization and control of helical robots for clearing superficial blood clots. APL bioengineering, 3(2):026104, 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kummer MP, Abbott JJ, Kratochvil BE, Borer R, Sengul A, and Nelson BJ. OctoMag: An electromagnetic system for 5-DOF wireless micromanipulation. IEEE Transactions on Robotics, 26 (6):1006–1017, December 2010. [Google Scholar]
- Liang R-H, Cheng K-Y, Su C-H, Weng C-T, Chen B-Y, and Yang D-N. Gausssense: Attachable stylus sensing using magnetic sensor grid. In ACM Symposium on User Interface Software and Technology, UIST ’; 12, pages 319–326, New York, NY, USA, 2012. [Google Scholar]
- Mahoney A and Abbott J. Five-degree-of-freedom manipulation of an untethered magnetic device in fluid using a single permanent magnet with application in stomach capsule endoscopy. The International Journal of Robotics Research, 35(1–3):129–147, 2016. [Google Scholar]
- More JJ. The levenberg-marquardt algorithm: implementation and theory. In Watson GA, editor, Numerical Analysis, volume 630 of Lecture Notes in Mathematics, pages 105–116. Springer, 1978. [Google Scholar]
- Morita E, Ohtsuka N, Shindo Y, Nouda S, Kuramoto T, Inoue T, Murano M, Umegaki E, and Higuchi K. In vivo trial of a driving system for a self-propelling capsule endoscope using a magnetic field. Gastrointestinal Endoscopy, 72(4):836–840, 2010. [DOI] [PubMed] [Google Scholar]
- Natali CD, Beccani M, and Valdastri P. Real-time pose detection for magnetic medical devices. IEEE Transactions on Magnetics, 49(7):3524–3527, 2013. [Google Scholar]
- Natali CD, Beccani M, Simaan N, and Valdastri P. Jacobian-based iterative method for magnetic localization in robotic capsule endoscopy. IEEE Transactions on Robotics, 32(2):327–338, 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pittiglio G, Barducci L, Martin JW, Norton JC, Avizzano CA, Obstein KL, and Valdastri P. Magnetic levitation for soft-tethered capsule colonoscopy actuated with a single permanent magnet: a dynamic control approach. IEEE Robotics and Automation Letters, 4(2):1224–1231, 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Popek KM, Schmid T, and Abbott JJ. Six-degree-of-freedom localization of an untethered magnetic capsule using a single rotating magnetic dipole. IEEE Robotics and Automation Letters, 2(1):305–312, 2017a. [Google Scholar]
- Popek KM, Tucker H, and Abbott JJ. First demonstration of simultaneous localization and propulsion of a magnetic capsule in a lumen using a single rotating magnet. In IEEE International Conference on Robotics and Automation, pages 1154–1160, 2017b. [Google Scholar]
- Quaglia C, Buselli E, III RJW, Valdastri P, Menciassi A, and Dario P. An endoscopic capsule robot: A meso-scale engineering case study. Journal of Micromechanics and Microengineering, 19 (10):105007, 2009. [Google Scholar]
- Rey J, Ogata H, Hosoe N, Ohtsuka K, Ogata N, Ikeda K, Aihara H, Pangtay I, Hibi T, Kudo S, and Tajiri H. Feasibility of stomach exploration with a guided capsule endoscope. Endoscopy, 42(7):541–545, 2010. [DOI] [PubMed] [Google Scholar]
- Rey J, Ogata H, Hosoe N, Ohtsuka K, Ogata N, Ikeda K, Aihara H, Pangtay I, Hibi T, Kudo S, and Tajiri H. First blinded non-randomized comparative study of gastric examination with a magnetically guided capsule endoscope and standard videoendoscope. Gastrointestinal Endoscopy, 75(2):373–381, 2011. [DOI] [PubMed] [Google Scholar]
- Rupprecht T, Rupprecht C, Muldorfer S, Vieth M, and Zapke M. The physical basics of magnetic-guided capsule endoscopy of the stomach and results of a feasibility study in the porcine stomach. Endoscopy, 44(4):437, 2012. [DOI] [PubMed] [Google Scholar]
- Sendoh M, Ishiyama K, and Arai K-I. Fabrication of magnetic actuator for use in a capsule endoscope. IEEE Transactions on Magnetics, 39(5):3232–3234, 2003. [Google Scholar]
- Son D, Yim S, and Sitti M. A 5-D localization method for a magnetically manipulated untethered robot using a 2-D array of Hall-effect sensors. IEEE/ASME Transactions on Mechatronics, 21(2):708–716, April 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Son D, Gilbert H, and Sitti M. Magnetically actuated soft capsule endoscope for fine-needle biopsy. Soft Robotics, 7(1):10–21, 2020. [DOI] [PubMed] [Google Scholar]
- Sudo S, Segawa S, and Honda T. Magnetic swimming mechanism in a viscous liquid. Journal of Intelligent Materials and Structures, 17:729–736, 2006. [Google Scholar]
- Swain P, Toor A, Volke F, Keller J, Gerber J, Rabinovitz E, and Rothstein R. Remote magnetic manipulation of a wireless capsule endoscope in the esophagus and stomach of humans. Gastrointestinal Endoscopy, 71(7):1290–1293, 2010. [DOI] [PubMed] [Google Scholar]
- Taddese AZ, Slawinski PR, Pirotta M, Momi ED, Obstein KL, and Valdastri P. Enhanced real-time pose estimation for closed-loop robotic manipulation of magnetically actuated capsule endoscopes. International Journal of Robotics Research, 37(8): 890–911, 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tognarelli S, Castelli V, Ciuti G, Natali CD, Sinibaldi E, Dario P, and Menciassi A. Magnetic propulsion and ultrasound tracking of endovascular devices. Journal of Robotic Surgery, 6(1):5–12, March 2012. [DOI] [PubMed] [Google Scholar]
- Yesin KB, Vollmers K, and Nelson BJ. Modeling and control of untethered biomicrorobots in a fluidic environment using electromagnetic fields. International Journal of Robotics Research, 25: 527–536, 2006. [Google Scholar]