Using Nexus OSS repository in an isolated environment

‍

No internet access, no libraries?

In our workplace, like in many other big firms, government offices, and banks, the internal network is isolated from the outside world. Nothing goes out and nothing goes in. But, how can you conveniently work with open source libraries if you cannot use maven or npm (and other repositories) to fetch needed libraries?

Nexus (https://www.sonatype.com/product-nexus-repository), and other similar products can be installed in pairs, to bridge between networks. But AFAIK that topology can only work in synchronous mode and you cannot add your own asynchronous checks like using sandboxes or anti virus checks - and that can be a problem. Plus, we wanted to use only the free, open source version of nexus (OSS), as we try to rely on open source as much as possible.

So we came up with a solution for that, drawn here:

Basically, we act as a proxy to the internal Nexus, receiving its requests (initiated by developers issuing ‘mvn install’ or ‘npm install’ commands). Next, we pass the requests to the external network via IBM’s Data Power (which verifies the request, in case someone tries to exploit the end point - similar products exist as well). Data Power passes the request to an external proxy, with internet access. This external proxy downloads the required library, along with all its dependencies, and passes it on to the organizational file checking system (sandboxes, antivirus, etc.). If the file is approved it is copied into the internal network (via a vault or a similar mechanism). There, the internal proxy comes into live again, picks up those libraries and uploads them into Nexus. Then the developer can retry to fetch the libraries and...  voilà! The libraries are there and he can use them in his code.

Inner Proxy

The inner proxy is a Java Spring Boot application. It has two parts:

• The first part is a REST controller, exposing a GET url that accepts requests from Nexus. Using a servlet filter these requests are caught and sent to the DataPower (Listing 1).
• The second part watches a designated folder, to which the approved libraries are copied (wrapped in a zip file). It extracts the libraries from the zip and uploads them into Nexus (Listing 2).

Outer Proxy

The outer proxy is also a Java Spring Boot application. It accepts GET requests for the needed libraries. Then it downloads the libraries and their dependencies from maven or npm, and places them in a designated folder. The organizational file checking system will take the files from there ,check them, and if they are approved they will be moved into the inner network wrapped as a zip file (Listing 3).

That’s it! You are welcome to use a similar solution in your environment and start enjoying the wealth of open source libraries out there…

Listing 1 - Intercepting the requests

public class NexusInterceptingFilter extends DispatcherServlet {

private static final long serialVersionUID = 1L;

private static final String DP_URL_KEY = "dp_url";

private static final String SEPARATOR = "/";

protected static Logger logger = LoggerFactory.getLogger(NexusInterceptingFilter.class);

@Override

protected void doDispatch(HttpServletRequest request, HttpServletResponse response) throws Exception {

if (!(request instanceof ContentCachingRequestWrapper)) {

request = new ContentCachingRequestWrapper(request);

}

if (!(response instanceof ContentCachingResponseWrapper)) {

response = new ContentCachingResponseWrapper(response);

}

HandlerExecutionChain handler = getHandler(request);

try {

super.doDispatch(request, response);

} finally {

handleRequests(request, response, handler);

updateResponse(response);

}

}

private void handleRequests(HttpServletRequest requestToCache, HttpServletResponse responseToCache,

HandlerExecutionChain handler) {

logger.info(">>>>> Fetching: "+ requestToCache.getRequestURI());

// This is for java only

if (requestToCache.getRequestURI().endsWith(".pom")) {

// make the request for the lib ( etc. abbot/abbot/0.13.0/abbot-0.13.0.pom )

RestTemplate rest = new RestTemplate();

Environment env = (Environment)ApplicationContextProvider.getBeanByClass(Environment.class);

String dpUrl = env.getProperty(DP_URL_KEY);

String theUrl = dpUrl+"?libName="+getLibRequest(requestToCache.getRequestURI());

logger.info("The request: "+theUrl);

rest.getForObject(theUrl, Void.class);

}

}

private String getLibRequest(String requestURI) {

if (requestURI.endsWith(".pom")) {

String[] parts = requestURI.split(SEPARATOR);

String version = parts[parts.length - 2];

String artifactId = parts[parts.length - 3];

StringBuilder grp = new StringBuilder();

for(int i=2;i<parts.length - 3;i++) {

grp.append(parts[i]).append(".");

}

return grp.toString().substring(0,grp.length()-1)+SEPARATOR+

artifactId+SEPARATOR+version;

}

return null;

}

private void updateResponse(HttpServletResponse response) throws IOException {

ContentCachingResponseWrapper responseWrapper = WebUtils.getNativeResponse(response,

ContentCachingResponseWrapper.class);

responseWrapper.copyBodyToResponse();

}

}

Listing 2 - Upload to Nexus

@Component

public class ScheduledLibrariesReader {

private static final String MANIFEST = "manifest.txt";

@Value("${vaultFolder}")

private String vaultFolder;

@Value("${tempFolde}")

private String tempFolder;

@Value("${nexus_url}")

private String nexusUploadUrl;

protected static Logger logger = LoggerFactory.getLogger(ScheduledLibrariesReader.class);

@Scheduled(fixedRate = 60000)

public void readFiles() throws Exception {

// Read all files (= zip for java)

try (Stream<Path> paths = Files.walk(Paths.get(vaultFolder))) {

paths.filter(Files::isRegularFile).forEach(f -> {

try {

// open the zip to a temp folder we create

String theFolder = parseAndRename(f);

// upload all files from this dir, according to the manifest

upload(theFolder);

// remove zip and files

removeAll(theFolder, f);

} catch (Exception e) {

throw new RuntimeException(e);

}

});

}

}

private String parseAndRename(Path zipFile) throws Exception {

// output dir is under the temp folder in a new folder, based on the zip name

String dirName = tempFolder + "/"

+ zipFile.getFileName().toString().substring(0, zipFile.getFileName().toString().lastIndexOf("."));

File destDir = new File(dirName);

if (!destDir.exists()) {

destDir.mkdir();

}

byte[] buffer = new byte[1024];

ZipInputStream zis = new ZipInputStream(new FileInputStream(zipFile.toFile()));

ZipEntry zipEntry = zis.getNextEntry();

while (zipEntry != null) {

File newFile = new File(destDir, zipEntry);

FileOutputStream fos = new FileOutputStream(newFile);

int len;

while ((len = zis.read(buffer)) > 0) {

fos.write(buffer, 0, len);

}

fos.close();

zipEntry = zis.getNextEntry();

}

zis.closeEntry();

zis.close();

//

return dirName;

}

@SuppressWarnings("unchecked")

private void upload(String theFolder) throws Exception {

List<String> lines = Files.readAllLines(new File(theFolder+"/"+MANIFEST).toPath());

for (String line : lines) {

String[] parts = line.split(":"); // group/artifact/version//filename

ByteArrayOutputStream stdout = new ByteArrayOutputStream();

   ByteArrayOutputStream stderr = new ByteArrayOutputStream();

   Map map = new HashMap();

   map.put("file", new File(theFolder + "/" + parts[3]));

   CommandLine cmdLine = new CommandLine(MAVEN_LOC+"mvn.cmd");

   cmdLine.addArgument("deploy:deploy-file");

   cmdLine.addArgument("-DgroupId=" + parts[0]);

   cmdLine.addArgument("-DartifactId=" + parts[1]);

   cmdLine.addArgument("-Dversion=" + parts[2]);

   cmdLine.addArgument("-DgeneratePom=true");

   cmdLine.addArgument("-Dpackaging=jar");

   cmdLine.addArgument("-DrepositoryId=nexus");

   cmdLine.addArgument("-Durl="+nexusUploadUrl);

   cmdLine.addArgument("-Dfile=${file}");

   cmdLine.setSubstitutionMap(map);

   DefaultExecutor executor = new DefaultExecutor();  

   PumpStreamHandler streamHandler = new PumpStreamHandler(stdout, stderr);

   executor.setStreamHandler(streamHandler);

   DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();

   executor.execute(cmdLine, resultHandler);

   resultHandler.waitFor();

   logger.info("Exit code for uploading "+line+" is "+resultHandler.getExitValue());

   logger.info("Error is:\n"+stderr.toString());

   logger.info("Output is:\n"+stdout.toString());

}

}

Listing 3 - Downloading from Maven

@RestController

public class ProxyController {

private static final String MANIFEST = "manifest.txt";

private static final String SEPARATOR = "\\";

@Value("${vaultFolder:d:/Temp/repo}")

private String vaultFolder;

protected static Logger logger = LoggerFactory.getLogger(ProxyController.class);

@GetMapping("/fetch")

// Expecting groupId/artifactId/version

public void fetchLib(@RequestParam String libName) throws Exception {

// Currently supporting Maven and NPM

if (isMavenLib(libName)) {

List<Path> libs = fetchMavenLibs(libName);

moveToVault(libs);

} else {

// assume npm

fetchJsonLib(libName);

}

}

private void moveToVault(List<Path> libs) throws IOException {

StringBuilder manifest = new StringBuilder();

String zipFile = vaultFolder + "/nexus_" + System.currentTimeMillis() + ".zip";

byte[] buffer = new byte[1024];

ZipOutputStream zos = null;

FileInputStream fis = null;

try {

FileOutputStream fos = new FileOutputStream(zipFile);

zos = new ZipOutputStream(fos);

//

for (Path path : libs) {

manifest.append(getPartsString(path)).append("\n");

// add to zip

fis = new FileInputStream(path.toFile());

// begin writing a new ZIP entry, positions the stream to the start of the entry data

zos.putNextEntry(new ZipEntry(path.getFileName().toString()));

int length;

while ((length = fis.read(buffer)) > 0) {

zos.write(buffer, 0, length);

}

zos.closeEntry();

// close the InputStream

fis.close();

//Files.move(p, new File(vaultFolder + "/" + p.getFileName()).toPath());

}

// add the manifest to the zip

zos.putNextEntry(new ZipEntry(MANIFEST));

zos.write(manifest.toString().getBytes());

zos.closeEntry();

//

} finally {

zos.close();

}

}

private List<Path> fetchMavenLibs(String libName) {

logger.info("Fetching lib " + libName);

String groupId = getGroupId(libName);

String artifactId = getArtifactId(libName);

String version = getVersion(libName);

JkDependencySet deps = JkDependencySet.of().and(groupId + ":" + artifactId + ":" + version)

//   e.g. "com.threerings:tripleplay:1.4")

.withDefaultScopes(COMPILE_AND_RUNTIME);

JkDependencyResolver resolver = JkDependencyResolver.of(JkRepo.ofMavenCentral());

List<Path> libs = resolver.resolve(deps, RUNTIME).getFiles().getEntries();

logger.info("Downloaded lobs: {}", libs);

return libs;

}

}

‍