Sikuli and selenium integration and usage
Sikuli and Selenium are popular automation tools for testing and automating user interactions with software applications. Sikuli is an open-source tool designed primarily for image-based automation that utilizes image recognition to automate user interactions with a graphical user interface (GUI). While Selenium is commonly used for web-based automation.
This answer shows the integration of both Sikuli and Selenium to leverage their respective capabilities. Here's an overview:
Setup: We will set up the environment to import Sikuli and Selenium in our Java code file.
Usage: We will run the Java code to test Sikuli and Selenium integration.
Note: Sikuli is a GUI based automation tool, so we need a display monitor or virtual framebuffer for it to work. For this, we will use the virtual display environment of the Educative platform here.
Let's start by installing Sikuli and Selenium in our Debian-based Linux distribution.
Setup
To integrate Sikulix and Selenium, we will need the following packages to be installed on our system.
Java JDK
The
.jarfile of Sikulix (API or IDE)The
.jarfile of SeleniumChrome Webdriver
We can install the default Java Development Kit (JDK) using the following commands:
apt install default-jdk
We can use the following command to download the .jar file of Sikulixapi.
wget https://launchpad.net/sikuli/sikulix/2.0.5/+download/sikulixapi-2.0.5-lux.jar
Note: For this example, we are using version
2.0.5of the Sikulixapi. To choose another version, visit github page of Sikulix downlods.
We can use the following command to download the .jar file of Selenium.
wget https://github.com/SeleniumHQ/selenium/releases/download/selenium-4.10.0/selenium-server-4.10.0.jar
Note: For this example, we are using version
4.10.0of the Selenium server. To see other recommendations, visit the download page of the official Selenium website.
We can use the following commands to download and unzip Chromedriver for Selenium.
wget https://chromedriver.storage.googleapis.com/<VERSION>/chromedriver_linux64.zipunzip chromedriver_linux64.zip
In the above command <VERSION> represents the version of Chromedriver we want to download. In this example, we are using version 114.0.5735.90 of the Chrome browser. Therefore, we must replace it with the version number in the above command.
Note: The version of Chromedriver and the version of Chrome installed on the system must be compatible. To download the appropriate version of the Chromedriver, visit the WebDriver for Chrome page on the official Chromium website.
We can use the following command to check the installed version of Chrome on our system.
google-chrome --version
After downloading and moving these files into place, we can use Sikuli and Selenium in our Java code.
Usage
Let's first run the following Java code to see the output, and then we will understand the code line by line.
import org.sikuli.script.Screen;
import org.sikuli.script.FindFailed;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.chrome.ChromeDriver;
public class Main {
public static void main(String[] args) throws InterruptedException, FindFailed {
ChromeOptions options = new ChromeOptions();
options.addArguments("--no-sandbox");
WebDriver driver = new ChromeDriver(options);
driver.get("https://www.google.com");
driver.findElement(By.name("q")).sendKeys("Sikulix\n");
Screen s = new Screen();
s.wait("/usercode/images-button.png");
s.click();
Thread.sleep(5000);
driver.quit();
}
}Note: The output produced by the code above might be influenced by a slow internet connection. It is advisable to rerun the code in case you encounter an error or do not observe any output.
Explanation
Lines 1–7: We are importing the required modules of Sikuli and Selenium into our code file.
Lines 12–14: We are creating an instance of Chrome driver. Here the
--no-sandboxparameter is required to start Chrome on the Educative platform. This option may not be needed if we are running the code in a local environment.Line 16: We are navigating to the Google home page.
Line 17: We are entering
Sikulixin the search bar of Google by usingfindElementandsendKeysmethods of the Selenium web driver.Line 19: We are creating an instance of the
Screenmodule of the Sikuli package.Line 20: After Google loads the search results, we wait for the following image to appear on the screen.
Line 21: As the image is detected, we click on the "Images" button using the
clickmethod of theScreenmodule.Line 23: Added a 5-second delay before closing the browser to allow users to observe changes.
Line 24: We terminate the browser after executing the test code.
Free Resources