DEV Community

Cover image for How to develop an online code compiler using Java and Docker
Zakaria Maaraki
Zakaria Maaraki

Posted on • Updated on

How to develop an online code compiler using Java and Docker

In this tutorial, we are going to see an overview of how to create a simple and efficient online code compiler for competitive programming and coding interviews using Java (Spring Boot) and Docker.

Before starting you can find the source code of the project at the following link https://github.com/zakariamaaraki/RemoteCodeCompiler

Why Java and Docker?

Well for java it's a personal choice, I really like working with this language, but you can choose any other programming language you prefer.

To separate the execution environments from different source codes (to avoid malicious code affecting your machine) and to limit resources, we need to use virtual machines or containers, but what is the difference between these two choices?

Virtual machine vs container

Alt text

A virtual machine (VM) is an emulation of a computer system. Put simply, it makes it possible to run what appear to be many separate computers on hardware that is actually one computer.

With containers, instead of virtualizing the underlying computer like a virtual machine (VM), just the OS is virtualized.

Containers sit on top of a physical server and its host OS — typically Linux or Windows. Each container shares the host OS kernel and, usually, the binaries and libraries, too. Shared components are read-only.

So now it's clear that using containers it's a better choice for us as they are lighter than Vms, and docker is the most used containerization solution, that's why i chose docker for this project.

The API

Alt text

First, we need to design the API, let's say we want to provide an online compiler for 4 programming languages (Java, C, C++ and Python).
So the API should look like something like this :
Four controllers one for Java, one for C, one for C ++ and another for Python.
The call to these controllers is done through POST requests to the following urls :

  • localhost:8080/compiler/java
  • localhost:8080/compiler/c
  • localhost:8080/compiler/cpp
  • localhost:8080/compiler/python

As inputs, we are expecting 5 fields :

  • output : the expected output.
  • sourceCode : the source code in java, c, c++ or python.
  • timeLimit : the time limit in seconds that the source code must not exceed during its execution (must be between 0 and 15s).
  • memoryLimit : the memory limit in Mb that the source code must not exceed during its execution (must be between 0 and 1000Mb).
  • inputFile : inputs written in separate lines (optional).
// Python Compiler
    @RequestMapping(
            value = "python",
            method = RequestMethod.POST
    )
    public ResponseEntity<Object> compile_python(@RequestPart(value = "output", required = true) MultipartFile output,
                                            @RequestPart(value = "sourceCode", required = true) MultipartFile sourceCode,
                                            @RequestParam(value = "inputFile", required = false) MultipartFile inputFile,
                                            @RequestParam(value = "timeLimit", required = true) int timeLimit,
                                            @RequestParam(value = "memoryLimit", required = true) int memoryLimit
    ) throws Exception {
        return compiler(output, sourceCode, inputFile, timeLimit, memoryLimit, Langage.Python);
    }

    // C Compiler
    @RequestMapping(
            value = "c",
            method = RequestMethod.POST
    )
    public ResponseEntity<Object> compile_c(@RequestPart(value = "output", required = true) MultipartFile output,
                                          @RequestPart(value = "sourceCode", required = true) MultipartFile sourceCode,
                                          @RequestParam(value = "inputFile", required = false) MultipartFile inputFile,
                                          @RequestParam(value = "timeLimit", required = true) int timeLimit,
                                          @RequestParam(value = "memoryLimit", required = true) int memoryLimit
    ) throws Exception {
        return compiler(output, sourceCode, inputFile, timeLimit, memoryLimit, Langage.C);
    }

    // C++ Compiler
    @RequestMapping(
            value = "cpp",
            method = RequestMethod.POST
    )
    public ResponseEntity<Object> compile_cpp(@RequestPart(value = "output", required = true) MultipartFile output,
                                            @RequestPart(value = "sourceCode", required = true) MultipartFile sourceCode,
                                            @RequestParam(value = "inputFile", required = false) MultipartFile inputFile,
                                            @RequestParam(value = "timeLimit", required = true) int timeLimit,
                                            @RequestParam(value = "memoryLimit", required = true) int memoryLimit
    ) throws Exception {
        return compiler(output, sourceCode, inputFile, timeLimit, memoryLimit, Langage.Cpp);
    }

    // Java Compiler
    @RequestMapping(
            value = "java",
            method = RequestMethod.POST
    )
    public ResponseEntity<Object> compile_java(@RequestPart(value = "output", required = true) MultipartFile output,
                                          @RequestPart(value = "sourceCode", required = true) MultipartFile sourceCode,
                                          @RequestParam(value = "inputFile", required = false) MultipartFile inputFile,
                                          @RequestParam(value = "timeLimit", required = true) int timeLimit,
                                          @RequestParam(value = "memoryLimit", required = true) int memoryLimit
    ) throws Exception {
        return compiler(output, sourceCode, inputFile, timeLimit, memoryLimit, Langage.Java);
    }
Enter fullscreen mode Exit fullscreen mode

What type of response should the user expect?

Well, let's take a minute on this point. If you are doing competitive programming on platforms like Codeforces, Leetcode, or others you can see that there is 6 types of verdict (Accepted, Wrong answer, Compilation error, Runtime error, Out of memory error, and Time limit exceeded).

Compiling the source code inside a docker container

The idea here is to take the source code provided by the user and create a docker image depending on the language then run a container of this image to compile and execute the source code. Depending on the returning code of the container we can decide the verdict that we discussed before, but we need to make sure that the container doesn't exceed the time limit to avoid infinite executions, neither the memory limit (to avoid malicious code).

// Compile method
    private ResponseEntity<Object> compiler(
            MultipartFile output,
            MultipartFile sourceCode,
            MultipartFile inputFile,
            int timeLimit,
            int memoryLimit,
            Langage langage
    ) throws Exception {

        String folder = "utility";
        String file = "main";
        if(langage == Langage.C) {
            folder += "_c";
            file += ".c";
        } else if(langage == Langage.Java) {
            file += ".java";
        } else if(langage == Langage.Cpp) {
            folder += "_cpp";
            file += ".cpp";
        } else {
            folder += "_py";
            file += ".py";
        }

        if(memoryLimit < 0 || memoryLimit > 1000)
            return ResponseEntity
                    .badRequest()
                    .body("Error memoryLimit must be between 0Mb and 1000Mb");

        if(timeLimit < 0 || timeLimit > 15)
            return ResponseEntity
                    .badRequest()
                    .body("Error timeLimit must be between 0 Sec and 15 Sec");

        LocalDateTime date = LocalDateTime.now();

        createEntrypointFile(sourceCode, inputFile, timeLimit, memoryLimit, langage);

        logger.info("entrypoint.sh file has been created");

        saveUploadedFiles(sourceCode, folder + "/" + file);
        saveUploadedFiles(output, folder + "/" + output.getOriginalFilename());
        if(inputFile != null)
            saveUploadedFiles(inputFile, folder + "/" + inputFile.getOriginalFilename());
        logger.info("Files have been uploaded");

        String imageName = "compile" + new Date().getTime();

        logger.info("Building the docker image");
        String[] dockerCommand = new String[] {"docker", "image", "build", folder, "-t", imageName};
        ProcessBuilder probuilder = new ProcessBuilder(dockerCommand);
        Process process = probuilder.start();
        int status = process.waitFor();
        if(status == 0)
            logger.info("Docker image has been built");
        else
            logger.info("Error while building image");

        logger.info("Running the container");
        dockerCommand = new String[] {"docker", "run", "--rm", imageName};
        probuilder = new ProcessBuilder(dockerCommand);
        process = probuilder.start();
        status = process.waitFor();
        logger.info("End of the execution of the container");

        BufferedReader outputReader = new BufferedReader(new InputStreamReader(output.getInputStream()));
        StringBuilder outputBuilder = new StringBuilder();
        BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
        StringBuilder builder = new StringBuilder();

        boolean ans = runCode(outputReader, outputBuilder, reader, builder);

        String result = builder.toString();

        // delete files
        deleteFile(folder, file);
        new File(folder + "/" + output.getOriginalFilename()).delete();
        if(inputFile != null)
            new File(folder + "/" + inputFile.getOriginalFilename()).delete();
        logger.info("files have been deleted");

        String statusResponse = statusResponse(status, ans);

        logger.info("Status response is " + statusResponse);

        return ResponseEntity
                .status(HttpStatus.OK)
                .body(new Response(builder.toString(), outputBuilder.toString(), statusResponse, ans, date));
    }
Enter fullscreen mode Exit fullscreen mode

The entrypoint of the container

To configure the entrypoint of the docker image, we create during the execution an entrypoint file that contains the time limit, memory limit and the execution command.

This is an example of how to generate the entrypoint.sh file for java execution :

// create Java entrypoint.sh file
    private void createJavaEntrypointFile(String fileName, int timeLimit, int memoryLimit, MultipartFile inputFile) {
        String executionCommand = inputFile == null
                ? "timeout --signal=SIGTERM " + timeLimit + " java " + fileName.substring(0,fileName.length() - 5) + "\n"
                : "timeout --signal=SIGTERM " + timeLimit + " java " + fileName.substring(0,fileName.length() - 5) + " < " + inputFile.getOriginalFilename() + "\n";
        String content = "#!/usr/bin/env bash\n" +
                "mv main.java " + fileName+ "\n" +
                "javac " + fileName + "\n" +
                "ret=$?\n" +
                "if [ $ret -ne 0 ]\n" +
                "then\n" +
                "  exit 2\n" +
                "fi\n" +
                "ulimit -s " + memoryLimit + "\n" +
                 executionCommand +
                "exit $?\n";
        OutputStream os = null;
        try {
            os = new FileOutputStream(new File("utility/entrypoint.sh"));
            os.write(content.getBytes(), 0, content.length());
        } catch (IOException e) {
            e.printStackTrace();
        }finally{
            try {
                os.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
Enter fullscreen mode Exit fullscreen mode

Talk is cheap, show me the code 😃

This was just an overview, instead of explaining how to code all this, I suggest you to visit the link of my Github repository https://github.com/zakariamaaraki/RemoteCodeCompiler

If you have any questions please feel free to ask them !

Discussion (7)

Collapse
xedrtuy profile image
aldwnesx

!/usr/bin/env bash

ulimit -s 5
timeout --signal=SIGTERM 5 python3 test.py
exit $?

the above code doesn't seem to stop the python script. I tried with an infinite loop code in test.py

Collapse
zakariamaaraki profile image
Zakaria Maaraki Author

Hello, i'll look at it

Collapse
zakariamaaraki profile image
Zakaria Maaraki Author

Hello, fixed !
Now we destroy the container if it gets stuck for some reason after a timeout.
check the last version github.com/zakariamaaraki/RemoteCo...

Collapse
chandradharrao1 profile image
Chandradhar Rao

Really nice tutorial!

Collapse
zakariamaaraki profile image
Zakaria Maaraki Author

Thank you !

Collapse
akshithgithub profile image
Info Comment hidden by post author - thread only accessible via permalink
Akshith

Facing an issue,

13 170.2 [INFO] Changes detected - recompiling the module!

13 170.3 [INFO] Compiling 6 source files to /compiler/target/classes

13 174.8 [INFO] ------------------------------------------------------------------------

13 174.8 [INFO] BUILD FAILURE

13 174.8 [INFO] ------------------------------------------------------------------------

13 174.8 [INFO] Total time: 02:53 min

13 174.8 [INFO] Finished at: 2021-06-04T07:02:58Z

13 174.8 [INFO] ------------------------------------------------------------------------

13 174.8 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project compiler: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @1b30a54e -> [Help 1]

13 174.8 [ERROR]

13 174.8 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

13 174.8 [ERROR] Re-run Maven using the -X switch to enable full debug logging.

13 174.8 [ERROR]

13 174.8 [ERROR] For more information about the errors and possible solutions, please read the following articles:

13 174.8 [ERROR] [Help 1] cwiki.apache.org/confluence/displa...

Collapse
xedrtuy profile image
Info Comment hidden by post author - thread only accessible via permalink
aldwnesx

entrypoint.sh with time limit and memory limit does not work

Some comments have been hidden by the post's author - find out more