Open Access
Add to BibSonomyAbstract
Complementary Metal-Oxide-Semiconductor (CMOS) image sensors are widely used in mobile-phone and cameras. Hence, it is attractive if these image sensors can be used as the visible light communication (VLC) receivers (Rxs). However, using these CMOS image sensors are challenging. In this work, we propose and demonstrate a VLC link using mobile-phone camera with data rate higher than frame rate of the CMOS image sensor. We first discuss and analyze the features of using CMOS image sensor as VLC Rx, including the rolling shutter effect, overlapping of exposure time of each row of pixels, frame-to-frame processing time gap, and also the image sensor “blooming” effect. Then, we describe the procedure of synchronization and demodulation. This includes file format conversion, grayscale conversion, column matrix selection avoiding blooming, polynomial fitting for threshold location. Finally, the evaluation of bit-error-rate (BER) is performed satisfying the forward error correction (FEC) limit.
© 2015 Optical Society of America
1. Introduction
Visible light communication (VLC), also known as optical wireless communication (OWC) or Light-Fidelity (Li-Fi) has gained increasing attention recently [1–6]. As there is a shortage of radio frequency (RF) spectra in traditional wireless communications, VLC can be a supplementary form. Some researchers also believe that VLC can serve as indoor wireless communications in future 5G systems. Light emitting diode (LED) can be a promising transmitter (Tx) for VLC, since it can be modulated at a relatively high speed when compared with traditional light sources. Besides, it has good light quality, low power consumption, long lifetime, and it can be integrated with the lighting system. Hence little extra cost is needed to provide the VLC. However, the white-light phosphor-based LED has a slow response to ~1 MHz. Recently many works have been performed to increase the modulation speed of the LED Tx, such as using equalizers [7–9], or advanced modulation formats [10].
Image sensors or cameras have been widely used in our daily lives. Nearly all mobile-phones available in the market nowadays have embedded Complementary Metal-Oxide-Semiconductor (CMOS) cameras allowing users to take photos and videos. Hence, it is attractive if these already existing mobile-phone or vehicle image sensors can be used as the VLC receivers (Rxs). However, using the CMOS image sensor as VLC Rx is challenging since the VLC data rate will be significantly limited by the image sensor frame rate. Besides, signal data loss will occur due to the relatively long processing time gap in frame-to-frame [11] proposed using the rolling shutter effect of the CMOS image sensor to increase the VLC data rate; however, our work provides detail discussion of the features of CMOS sensors, including the overlapping of exposure time of each row of pixels, frame-to-frame processing time gap, and also the image sensor “blooming” effect. Besides, in our proposed scheme, a column matrix of grayscale will be selected just away from the center of the blooming. This can provide distinguishable grayscale value even at very high blooming effect. Bit-error-rate (BER) analysis is also provided [12] demonstrated a special image sensor with specific pixels for imaging and specific pixels for high speed VLC; however, this sensor requires relatively complicated fabrication process, and may not be available as embedded device in mobile-phone and vehicle camera in the near future.
In this work, we propose and demonstrate a VLC link using mobile-phone camera with data rate higher than frame rate of the CMOS image sensor. We first discuss and analyze the features of using CMOS image sensor as VLC Rx, including the rolling shutter effect, overlapping of exposure time of each row of pixels, frame-to-frame processing time gap, and also the image sensor “blooming” effect. Then, we describe the procedure of synchronization and demodulation. This includes file format conversion, grayscale conversion, column matrix selection avoiding blooming, polynomial fitting for threshold location. The evaluation of BER is performed. Although the data rate is 1.68 kbit/s in the proof-of-concept demonstration; we believe that this data rate may enough for sending secure identification (ID) or authorization information in wireless communication; or sending position information in indoor navigation system.
2. Features of using mobile-phone camera as VLC Rx
Most CMOS image sensors embedded in the mobile-phone is operated using rolling shutter. This means that each row of pixels in the CMOS sensor is activated sequentially, and it does not capture the entire image at once (known as global shutter), as shown in Fig. 1. Although this can be an undesirable effect for taking video of a moving object; it can be beneficial for the VLC since the data rate can be much faster than the frame rate (frame per second; fps). Two scenarios can happen. If the VLC transmitting rate is slower than the image sensor frame rate (~30 fps), one or several image(s) can represent a logic bit; hence the effect of rolling shutter cannot be clearly observed. If the VLC transmitting rate is higher than the image sensor frame rate, bright and dark fringes can be observed in one image frame due to the rolling shutter effect. Hence one image can represent many logic bits. As shown in Fig. 1(a), if the ideal CMOS sensor exists, each row of pixels is activated one by one, then in principle, assuming the visible light detected by the image sensor has high enough signal-to-noise ratio (SNR), each row of pixels can represent one logic bit. However, in the real case, the row of pixels is activated without waiting for the previous row to complete the exposure. Hence, there is an overlapping of exposure time of each row as shown in Fig. 1(b). In this case, the effective number of bits represent in each image frame is reduced. In addition, as also shown in Fig. 1(b), when all the rows of pixels are activated, there is a frame-to-frame processing time gap, which is needed to merge different rows together to form a single image. As the Tx is continuously sending data, there will be a data loss during this frame-to-frame processing time gap at the Rx. Furthermore, if the received visible signal is too strong, image sensor “blooming” will occur. This is due to the overflow of charge from the saturated pixels into the neighboring pixels. Hence, the contract ratio between the bright and dark fringes will decrease. The blooming effect will be further discussed in Section 3.
Fig. 1 Schematic diagrams to show the idea of rolling shutter effect in CMOS image sensor, in (a) expected ideal case and (b) real case.
3. Experiment, results and discussion
Figure 2 shows the experimental setup of the proposed VLC link using mobile-phone camera as Rx. The transmitting data is first programmed using Matlab in a computer, which is then transferred to an arbitrary waveform generator (AWG) to perform the digit-to-analog (DAC) conversion. The AWG (Tektronix, AFG 3252C) has a sampling rate of 2 GSample/s and bandwidth of 240 MHz. The data signal is then emitted by a single white-light LED (Cree XLamp XR-E) with modulation bandwidth of about 1 MHz. The visible signal is received by a mobile-phone camera. The recorded movie by the mobile-phone is then demodulated by a Matlab program. As this transmission is basically an asynchronized link, it is very important to develop a synchronization scheme to recover the clock within each image.
As mentioned in Section 2, one feature of the CMOS Rx is the frame-to-frame processing time gap. As the LED Tx is continuously sending data, there will be a data loss during this frame-to-frame processing time gap. In the experiment, a Samsung GT-S7500 mobile-phone is used. The CMOS pixels are 480 x 640 and the recoding frame rate is 28 fps. We also measure the frame-to-frame time gap is about 14.29 ms, which is about 40% of the frame duration. During this frame-to-frame processing time gap, the image sensor is completely “blind”; hence each packet consists of two sets of headers and payloads in order to make sure each image contain enough header information for synchronization.
Figure 3 shows the structure of the transmitting data packet, in which each packet consists of two sets of headers and payloads. Different brand mobile-phones have different frame-to-frame processing time gaps. The data rate can be enhanced if more advanced mobile-phone can be used. As shown in Fig. 3, the header consists of 4 bits periodic signal using Manchester coding. This is enough for our clock recovery. In order to avoid the error propagation in the Manchester coding, on-off keying (OOK) format is used in the payload data. The payload consists of 60 bits or 30 bits. The image frame showing bright and dark fringes representing the packet header and payload is also included in Fig. 3.
Fig. 3 Structure of the data packet, in which each packet consists of two sets of headers and payloads. Image frame showing bright and dark fringes representing the packet header and payload is also included.
The procedure of synchronization and demodulation is shown in Fig. 4(a). First, the raw video file is converted into proper format, such as MPG file, which can be effectively processed in Matlab. Then each image file will be converted into grayscale format. In grayscale format, each row of pixels is represented from 0 to 255 grayscale levels, in which 0 represents completely dark fringe while 255 represents completely bright fringe. As discussed in Section 2, due to the overlapping of exposure time of each row and also the blooming effect (overflow of charge from the saturated pixels into the neighboring pixels), many other levels between 0 and 255 can be observed in each image frame.
Fig. 4 (a) Flow chat showing the procedure of the demodulation process. (b) One image frame illustrating the blooming at the center of the image.
Figure 4(b) show one image frame after the grayscale conversion. A column will be selected for the demodulation. However, as clearly shown in Fig. 4(b), in the center column (yellow vertical line) blooming occurs, and the contrast between bright and dark fringes decreases. Hence, the header information cannot be recovered. A column matrix of grayscale will be selected just away from the center of the blooming. In the experiment, every 60 rows of pixels are grouped (red line) in the column matrix selection. When the column matrix is selected, a proper threshold level should be set in order to recover the logic 1 and 0. As shown in Fig. 5, the blue curve represents the column matrix grayscale values from 0 to 255. Then a 2nd ordered polynomial fitting is applied to the column matrix, and the red curve shown in Fig. 5 can be obtained. The 2nd ordered polynomial fitting can be used as a threshold to distinguish the logic 1 and 0. As the header is encoded using Manchester coding, much narrower periodic bring and dark fringes can be easily distinguished from the payload OOK data, which consists of much wider fringes. Hence the header can be easily located in the image frame. Then, by counting the transition rates of these narrow fringes, the clock and sampling rate information can be obtained. Finally, the BER of the payload can be evaluated.
Fig. 5 Column matrix of grayscale values (0 to 255) (blue curve) and the 2nd order polynomial fitting used as the threshold to distinguish the logic 1 and 0 (red curve).
After applying the file format conversion, grayscale conversion, column matrix selection avoiding blooming, polynomial fitting for threshold location; the data logic can be recovered. Figures 6(a) and 6(b) show the image frame received at different illuminance to illustrate the successful logic bit recovery. In Fig. 6, logic 1 will be converted to grayscale value 255, while logic 0 will be converted to grayscale value 0. This vertical column of converted grayscale level is then pasted at the center of the image frame for illustration. We can see that the header information can be easily observed (narrower periodic bright and dark fringes) and the blooming effect is mitigated with enhanced bright and dark fringes contrast ratio.
Fig. 6 Image frame received at different illuminance. The vertical column of converted grayscale level is pasted at the center showing the blooming effect is mitigated with enhanced contrast ratio.
Finally, we evaluate the BER performance as shown in Fig. 7. Due to the overlapping of exposure time of each row in the rolling shutter effect as discussed in Section 2, the effective number of bits represented in each image frame is reduced. When using the 60-bit payload, the effective data rate (removing the header and duplicated payload) of the proposed VLC system is about 60 bits/frame x 28 fps = 1.68 kbit/s. When using the 30-bit payload, the BER performance is enhanced due to doubling the pixel row resolution. It is worth to point out that as high-end mobile phones with better image senor resolution (1080 rows) may be common due to cost-down in the future, the data rate can be enhanced. We also believe that the data rate in this proof-of-concept demonstration may be enough for sending secure ID or authorization information wirelessly; or sending position information in indoor navigation system. It is also worth to mention that although the proposed schemes and procedures here could be applied to different mobile-phone, the obtained results may be different due to the mobile-phone hardware and software changes.
4. Conclusion
In this work, we proposed and demonstrated a VLC link using mobile-phone camera with data rate higher than frame rate of the CMOS image sensor. We first discussed and analyzed the features of using CMOS image sensor as VLC Rx, including the rolling shutter effect, overlapping of exposure time of each row of pixels, frame-to-frame processing time gap, and also the image sensor blooming effect. We described our procedure of synchronization and demodulation, including file format conversion, grayscale conversion, column matrix selection avoiding blooming, polynomial fitting for threshold location. The demonstrated data rate was 1.68 kbit/s. It is believed that the data rate in this proof-of-concept demonstration may be enough for sending secure ID or authorization information wirelessly; or sending position information in indoor navigation system.
Acknowledgments
This work was supported by Ministry of Science and Technology, Taiwan, ROC, MOST- 104-2628-E-009-011-MY3, MOST-103-2221-E-009-030-MY3, Aim for the Top University Plan, Taiwan, and Ministry of Education, Taiwan.
References and links
1. C. W. Chow, C. H. Yeh, Y. Liu, and Y. F. Liu, “Digital signal processing for light emitting diode based visible light communication,” IEEE Photonics Soc. Newslett. 26, 9–13 (2012).
2. H. H. Lu, Y. P. Lin, P. Y. Wu, C. Y. Chen, M. C. Chen, and T. W. Jhang, “A multiple-input-multiple-output visible light communication system based on VCSELs and spatial light modulators,” Opt. Express 22(3), 3468–3474 (2014). [CrossRef] [PubMed]
3. W. Y. Lin, C. Y. Chen, H. H. Lu, C. H. Chang, Y. P. Lin, H. C. Lin, and H. W. Wu, “10m/500 Mbps WDM visible light communication systems,” Opt. Express 20(9), 9919–9924 (2012). [CrossRef] [PubMed]
4. Z. Wang, C. Yu, W. D. Zhong, J. Chen, and W. Chen, “Performance of a novel LED lamp arrangement to reduce SNR fluctuation for multi-user visible light communication systems,” Opt. Express 20(4), 4564–4573 (2012). [CrossRef] [PubMed]
5. C. H. Chang, C. Y. Li, H. H. Lu, C. Y. Lin, J. H. Chen, Z. W. Wan, and C. J. Cheng, “A 100-Gb/s multiple-input multiple-output visible laser light communication system,” J. Lightwave Technol. 32(24), 4723–4729 (2014). [CrossRef]
6. Z. Wang, W. D. Zhong, C. Yu, J. Chen, C. P. Francois, and W. Chen, “Performance of dimming control scheme in visible light communication system,” Opt. Express 20(17), 18861–18868 (2012). [CrossRef] [PubMed]
7. H. Le-Minh, D. O’Brien, G. Faulkner, L. Zeng, K. Lee, D. Jung, and Y. Oh, “High-speed visible light communications using multiple-resonant equalization,” IEEE Photon. Technol. Lett. 20(14), 1243–1245 (2008). [CrossRef]
8. J. Y. Sung, C. W. Chow, and C. H. Yeh, “Is blue optical filter necessary in high speed phosphor-based white light LED visible light communications?” Opt. Express 22(17), 20646–20651 (2014). [CrossRef] [PubMed]
9. C. H. Yeh, C. W. Chow, H. Y. Chen, J. Chen, and Y. L. Liu, “Adaptive 84.44 - 190 Mbit/s phosphor-LED wireless communication utilizing no blue filter at practical transmission distance,” Opt. Express 22(8), 9783–9788 (2014). [CrossRef] [PubMed]
10. J. Y. Sung, C. W. Chow, and C. H. Yeh, “Dimming-discrete-multi-tone (DMT) for simultaneous color control and high speed visible light communication,” Opt. Express 22(7), 7538–7543 (2014). [CrossRef] [PubMed]
11. C. Danakis, M. Afgani, G. Povey, I. Underwood, and H. Haas, “Using a CMOS camera sensor for visible light communication,” in Proc. IEEE Workshop on Opt. Wireless Comm. (OWC, 2012), pp. 1244–1248. [CrossRef]
12. I. Takai, S. Ito, K. Yasutomi, K. Kagawa, M. Andoh, and S. Kawahito, “LED and CMOS image sensor based optical wireless communication system for automotive applications,” IEEE Photonics J. 5(5), 6801418 (2013). [CrossRef]
