Release 1.7

This extension allows operators to specify what IRR source to use when expanding an AS-SET.

For example: `bgpq4 RIPE::AS15562:AS-SNIJDERS`

The notation is also used in databases like PeeringDB.

Authored-by: James Bensley <jwbensley@gmail.com>

.github/images/rockylinux/rockylinux:9.Dockerfile

@@ -0,0 +1 @@
+../centos.Dockerfile

.github/images/ubuntu:jammy.Dockerfile

@@ -0,0 +1 @@
+debian.Dockerfile

.github/workflows/matrixbuild.yml

@@ -12,17 +12,19 @@ jobs:
           - debian:bookworm
           - debian:bullseye
           - debian:buster
+          - ubuntu:jammy
           - ubuntu:focal
           - ubuntu:bionic
+          - fedora/fedora:37
           - fedora/fedora:36
           - fedora/fedora:35
-          - fedora/fedora:34
           - centos/centos:stream9
           - centos/centos:stream8
           - centos/centos:7
+          - rockylinux/rockylinux:9
           - rockylinux/rockylinux:8
           - alpine:edge
-          - alpine:3.15
+          - alpine:3.16
     steps:
     - uses: actions/checkout@v1
     - name: Run build on ${{matrix.dockerenv}}

CHANGES

@@ -1,3 +1,9 @@
+1.7 (2022-11-03)
+    - Support SOURCE:: syntax (contributed by James Bensley)
+
+1.6 (2022-09-07)
+    - Fix a bug in address prefix range parsing
+
 1.5 (2022-07-25)
     - Add support for the new Junos as-path-origins feature
 

README.md

@@ -27,10 +27,10 @@
 
 The
 **bgpq4**
-utility used to generate configurations (prefix-lists, extended
+utility is used to generate configurations (prefix-lists, extended
 access-lists, policy-statement terms and as-path lists) based on IRR data.
 
-The options are as follows:
+It's options are as follows:
 
 **-4**
 
@@ -162,7 +162,7 @@ The options are as follows:
 
 **-W** *len*
 
-> generate as-path strings of no more than len items (use 0 for inifinity).
+> generate as-path strings of no more than len items (use 0 for infinity).
 
 **-U**
 
@@ -342,28 +342,70 @@ be in one line (sometimes it makes sense):
 
 # NOTES ON SOURCES
 
-By default
-*bgpq4*
-trusts to data from all databases mirrored into NTT's IRR service.
-Unfortunately, not all these databases are equal in how much can we
-trust their data.
+By default *bgpq4* trusts data from all databases mirrored into NTT's IRR service.
+Unfortunately, not all these databases are equal in how much can we trust their 
+data.
 RIR maintained databases (AFRINIC, ARIN, APNIC, LACNIC and RIPE)
-shall be trusted more than the others because they are indeed have the
-knowledge about which address space allocated to this or that ASn,
-other databases lack this knowledge and can (and, actually, do) contain
-some stale data: noone but RIRs care to remove outdated route-objects
-when address space revoked from one ASn and allocated to another.
-In order to keep their filters both compact and actual,
-*bgpq4 users*
-are encouraged to use '-S' flag to limit database sources to only
-ones they trust.
+shall be trusted more than the others because they have the knowledge about 
+which address space is allocated to each ASN, other databases lack this 
+knowledge and can (and actually do) contain some stale data: nobody but RIRs 
+care to remove outdated route-objects when address space is revoked from one 
+ASN and allocated to another. In order to keep their filters both compact and 
+current, *bgpq4 users* are encouraged to use one of two method to limit 
+database sources to only ones they trust.
+
+One option is to use the '-S' flag. This limits all queries to a specific data 
+source. For example, the following command tells IIRd to only use data from 
+the RIPE RIR DB to build the prefix list for the AS-SET:
+
+	$./bgpq4 -S RIPE AS-VOSTRON
+	no ip prefix-list NN
+	ip prefix-list NN permit 89.21.224.0/19
+	ip prefix-list NN permit 134.0.64.0/21
+
+Be aware though, than an AS-SET may contain members from other data sources.
+In this case IRRd won't respond to the bgpq4 query will all the prefixes in the 
+AS-SET tree. Make sure to use the '-S' flag with all the data sources required 
+for the AS-SET being expanded:
+
+	$./bgpq4 -S RIPE,ARIN AS-VOSTRON
+	no ip prefix-list NN
+	ip prefix-list NN permit 89.21.224.0/19
+	ip prefix-list NN permit 134.0.64.0/21
+	ip prefix-list NN permit 208.86.232.0/24
+	ip prefix-list NN permit 208.86.233.0/24
+	ip prefix-list NN permit 208.86.234.0/24
+	ip prefix-list NN permit 208.86.235.0/24
+
+The other option is to specify a source for an AS-SET or Route Set using the 
+"::" notation. When bgpq4 detects this, it will look for "::" in the specified 
+AS-SET or RS on the CLI, and in all members of the AS-SET/RS, and for each 
+member with a data source specified in "::" format, it will set the IRRd data 
+source to the given value, query the AS-SET/RS, then reset the data sources back
+ to the default list for the next object in the tree.
+
+	$./bgpq4 RIPE::AS-VOSTRON
+	no ip prefix-list NN
+	ip prefix-list NN permit 89.21.224.0/19
+	ip prefix-list NN permit 134.0.64.0/21
+	ip prefix-list NN permit 208.86.232.0/22
+	ip prefix-list NN permit 208.86.232.0/24
+	ip prefix-list NN permit 208.86.233.0/24
+	ip prefix-list NN permit 208.86.234.0/24
+	ip prefix-list NN permit 208.86.235.0/24
+
+In comparison to the '-S' flag, this method return all the prefixes under the 
+AS-SET, but the root of the tree "AS-VOSTRON" was queries from RIPE only. None 
+of the member objects used the "::" notation so they were queries from the 
+default source list (which is all sources).
+
 
 General recommendations:
 
 Use minimal set of RIR databases (only those in which you and your
 customers have registered route-objects).
 
-Avoid using ARIN-NONAUTH and RIPE-NONAUTH as trusted source: these records
+Avoid using ARIN-NONAUTH and RIPE-NONAUTH as trusted sources: these records
 were created in database but for address space allocated to different RIR,
 so the NONAUTH databases have no chance to confirm validity of this route
 object.
@@ -380,6 +422,9 @@ object.
 	ip prefix-list NN permit 45.65.184.0/22
 	[...]
 
+When known, use the "::" notation to speicy the authortative data source for 
+an AS-SET or RS instead of the -S flag.
+
 # PERFORMANCE
 
 To improve \`bgpq4\` performance when expanding extra-large AS-SETs you
@@ -430,7 +475,7 @@ In order to create a distribution archive, run:
 When everything is OK,
 **bgpq4**
 generates access-list to standard output and exits with status == 0.
-In case of errors they are printed to stderr and program exits with
+In case of errors they are printed to stderr and the program exits with
 non-zero status.
 
 # AUTHORS

VERSION

@@ -1 +1 @@
-1.5
+1.7

bgpq4.8

@@ -125,12 +125,12 @@ use specified sources only (recommended: RPKI,AFRINIC,APNIC,ARIN,LACNIC,RIPE).
 generate as-sets for OpenBGPd, BIRD and JSON formats.
 .It Fl T
 disable pipelining (not recommended).
-.It Fl W Ar len
-generate as-path strings of no more than len items (use 0 for inifinity).
 .It Fl U
 generate config for Huawei devices (Cisco IOS by default)
 .It Fl u
 generate config for Huawei devices in XPL format (Cisco IOS by default)
+.It Fl W Ar len
+generate as-path strings of no more than len items (use 0 for inifinity).
 .It Fl X
 generate config for Cisco IOS XR devices (plain IOS by default).
 .It Fl z

expander.c

@@ -86,6 +86,7 @@ bgpq_expander_init(struct bgpq_expander *b, int af)
 
 	b->family = af;
 	b->sources = "";
+	b->usesource = 0;
 	b->name = "NN";
 	b->aswidth = 8;
 	b->identify = 1;
@@ -152,7 +153,7 @@ bgpq_expander_add_already(struct bgpq_expander *b, char *rs)
 	lkey.text = rs;
 
 	if (RB_FIND(tentree, &b->already, &lkey))
-		return 1;
+		return 0;
 
 	le = sx_tentry_new(rs);
 
@@ -169,7 +170,7 @@ bgpq_expander_add_stop(struct bgpq_expander *b, char *rs)
 	lkey.text = rs;
 
 	if (RB_FIND(tentree, &b->stoplist, &lkey))
-		return 1;
+		return 0;
 
 	le = sx_tentry_new(rs);
 
@@ -196,11 +197,13 @@ bgpq_expander_add_as(struct bgpq_expander *b, char *as)
 	}
 
 	if (!expand_special_asn &&
-	    ((asno  >= 4200000000ul) || (asno >= 64496 && asno <= 65551)))
+	    ((asno  >= 4200000000ul) || (asno >= 64496 && asno <= 65551))) {
+		sx_report(SX_ERROR,"Invalid AS number: %u\n", asno);
 		return 0;
+	}
 
 	if ((asne = malloc(sizeof(struct asn_entry))) == NULL)
-		err(1, NULL);
+		sx_report(SX_FATAL, "malloc failed for asn\n");
 
 	asne->asn = asno;
 	RB_INSERT(asn_tree, &b->asnlist, asne);
@@ -229,7 +232,7 @@ bgpq_expander_add_prefix(struct bgpq_expander *b, char *prefix)
 	sx_radix_tree_insert(b->tree, p);
 
         if (p)
-		sx_prefix_destroy(p);
+		sx_prefix_free(p);
 
 	return 1;
 }
@@ -240,6 +243,49 @@ bgpq_expander_add_prefix_range(struct bgpq_expander *b, char *prefix)
 	return sx_prefix_range_parse(b->tree, b->family, b->maxlen, prefix);
 }
 
+char*
+bgpq_get_asset(char *object){
+	char *asset, *d;
+
+	d = strstr(object, "::");
+	if (d){
+		d += 2;
+	} else {
+		d = object;
+	}
+
+	if ((asset = calloc(1, 256)) == NULL)
+		sx_report(SX_FATAL, "calloc failed for asset\n");
+	memcpy(asset, d, strlen(object) - (d - object));
+	return asset;
+}
+
+char*
+bgpq_get_rset(char *object){
+	char *d = strstr(object, "::");
+	if (d){
+		d += 2;
+	} else {
+		d = object;
+	}
+
+	char *rset = (char*)calloc(1, 256);
+	memcpy(rset, d, strlen(object) - (d - object));
+	return rset;
+}
+
+char*
+bgpq_get_source(char *object){
+	char *d = strstr(object, "::");
+	if (d){
+		char *source = (char*)calloc(1, 256);
+		unsigned int slen = d - object;
+		memcpy(source, object, slen);
+		return source;
+	}
+	return NULL;
+}
+
 static int
 bgpq_expanded_macro(char *as, struct bgpq_expander *ex,
     struct request *req)
@@ -263,6 +309,7 @@ bgpq_expanded_macro_limit(char *as, struct bgpq_expander *b,
 {
 	if (!strncasecmp(as, "AS-", 3) || strchr(as, '-') || strchr(as, ':')) {
 		struct sx_tentry tkey = { .text = as };
+		char *source;
 
 		if (RB_FIND(tentree, &b->already, &tkey)) {
 			SX_DEBUG(debug_expander > 2, "%s is already expanding, "
@@ -281,14 +328,36 @@ bgpq_expanded_macro_limit(char *as, struct bgpq_expander *b,
 		    req->depth + 1 < b->maxdepth)) {
 			bgpq_expander_add_already(b, as);
 			if (pipelining) {
+				if (b->usesource) {
+					source = bgpq_get_source(as);
+					if (source){
+						bgpq_pipeline(b, NULL, NULL, "!s%s\n", source);
+						free(source);
+					} else {
+						bgpq_pipeline(
+							b, NULL, NULL, "!s%s\n", b->defaultsources
+						);
+					}
+				}
 				struct request *req1 = bgpq_pipeline(b,
 				    bgpq_expanded_macro_limit, NULL, "!i%s\n",
-				        as);
+				        bgpq_get_asset(as));
 				req1->depth = req->depth + 1;
 			} else {
+				if (b->usesource) {
+					source = bgpq_get_source(as);
+					if (source) {
+						bgpq_expand_irrd(b, NULL, NULL, "!s%s\n", source);
+						free(source);
+					} else {
+						bgpq_expand_irrd(
+							b, NULL, NULL, "!s%s\n", b->defaultsources
+						);
+					}
+				}
 				b->cdepth++;
 				bgpq_expand_irrd(b, bgpq_expanded_macro_limit,
-				    NULL, "!i%s\n", as);
+				    NULL, "!i%s\n", bgpq_get_asset(as));
 				b->cdepth--;
 			}
 		} else {
@@ -350,6 +419,73 @@ bgpq_expanded_v6prefix(char *prefix, struct bgpq_expander *ex,
 	return 1;
 }
 
+static char*
+bgpq_get_irrd_sources(int fd) {
+	int ret;
+	char *query = "!s-lc\n";
+	int qlen = strlen(query);
+	const unsigned int rsize = 256;
+	char *response = (char*)calloc(1, rsize);
+	char *sources = (char*)calloc(1, rsize);
+
+	SX_DEBUG(debug_expander, "Requesting source list %s", query);
+	if ((ret = write(fd, query, strlen(query))) != qlen) {
+		sx_report(SX_ERROR, "Partial write of query to "
+			"IRRd: %i bytes, %s\n", ret, strerror(errno));
+		close(fd);
+		free(sources);
+		free(response);
+		exit(1);
+	}
+
+	if (0 < read(fd, response, rsize)) {
+		SX_DEBUG(debug_expander, "Got answer %s", response);
+		if (*(response + strlen(response) - 2) != 'C') {
+			sx_report(SX_ERROR, "Invalid response "
+				"'%s': %s\n", response, query);
+			close(fd);
+			free(sources);
+			free(response);
+			exit(1);
+		}
+	} else {
+		sx_report(SX_ERROR, "failed to read sources\n");
+		close(fd);
+		free(sources);
+		free(response);
+		exit(1);
+	}
+
+	char *start = strchr(response, '\n');
+	if (start){
+		start += 1;
+		char *end = strchr(start, '\n');
+		if (!end) {
+			sx_report(SX_ERROR, "No 2nd newline in response '%s': %s\n",
+				response, query);
+			close(fd);
+			free(sources);
+			free(response);
+			exit(1);
+		}
+		unsigned int slen = end - start;
+		if (slen > rsize) {
+			memcpy(sources, start, rsize-1);
+		} else {
+			memcpy(sources, start, slen);
+		}
+	} else {
+		sx_report(SX_ERROR, "No 1st newline in response '%s': %s\n",
+			response, query);
+		close(fd);
+		free(sources);
+		free(response);
+		exit(1);
+	}
+	free(response);
+	return sources;
+}
+
 int bgpq_pipeline_dequeue(int fd, struct bgpq_expander *b);
 
 static struct request *
@@ -385,7 +521,7 @@ bgpq_pipeline(struct bgpq_expander *b,
     int (*callback)(char *, struct bgpq_expander *, struct request *),
     void *udata, char *fmt, ...)
 {
-	char			 request[128];
+	char			 request[256];
 	int			 ret;
 	struct request	*bp = NULL;
 	va_list			 ap;
@@ -402,7 +538,6 @@ bgpq_pipeline(struct bgpq_expander *b,
 		sx_report(SX_FATAL,"Unable to allocate %lu bytes: %s\n",
 		    (unsigned long)sizeof(struct request),
 		    strerror(errno));
-		exit(1);
 	}
 
 	if (STAILQ_EMPTY(&b->wq)) {
@@ -528,6 +663,7 @@ bgpq_read(struct bgpq_expander *b)
 {
 	static char response[256];
 	static int off = 0;
+	int rval = 1;
 
 	if (!STAILQ_EMPTY(&b->wq))
 		bgpq_write(b);
@@ -666,7 +802,7 @@ have3:
 					c[spn] = 0;
 				if (c[0] == 0)
 					break;
-				req->callback(c, b, req);
+				if (!req->callback(c, b, req)) rval = 0;
 				c += spn + 1;
 			}
 			assert(c == recvbuffer + togot);
@@ -674,24 +810,26 @@ have3:
 			free(recvbuffer);
 		} else if (response[0] == 'C') {
 			/* No data */
-			SX_DEBUG(debug_expander,"No data expanding %s\n",
+			SX_DEBUG(debug_expander,"No data expanding %s",
 			    req->request);
 			if (b->validate_asns)
 				bgpq_expander_invalidate_asn(b, req->request);
 		} else if (response[0] == 'D') {
-			/* .... */
-			SX_DEBUG(debug_expander,"Key not found expanding %s\n",
+			sx_report(SX_ERROR, "Key not found expanding %s",
 			    req->request);
 			if (b->validate_asns)
 				bgpq_expander_invalidate_asn(b, req->request);
+			rval = 0;
 		} else if (response[0] == 'E') {
-			sx_report(SX_ERROR, "Multiple keys expanding %s: %s\n",
+			sx_report(SX_ERROR, "Multiple keys expanding %s: %s",
 			    req->request, response);
+			rval = 0;
 		} else if ( response[0] == 'F') {
-			sx_report(SX_ERROR, "Error expanding %s: %s\n",
+			sx_report(SX_ERROR, "Error expanding %s: %s",
 			    req->request, response);
+			rval = 0;
 		} else {
-			sx_report(SX_ERROR,"Wrong reply: %s to %s\n", response,
+			sx_report(SX_ERROR,"Wrong reply: %s to %s", response,
 			    req->request);
 			exit(1);
 		}
@@ -708,7 +846,7 @@ have3:
 		request_free(req);
 	}
 
-	return 0;
+	return rval;
 }
 
 int
@@ -716,11 +854,12 @@ bgpq_expand_irrd(struct bgpq_expander *b,
     int (*callback)(char *, struct bgpq_expander *, struct request *),
     void *udata, char *fmt, ...)
 {
-	char			 request[128], response[128];
+	char			 request[256], response[256];
 	va_list			 ap;
 	ssize_t			 ret;
 	int			 off = 0;
 	struct request	*req;
+	int rval = 1;
 
 	va_start(ap, fmt);
 	vsnprintf(request, sizeof(request), fmt, ap);
@@ -730,8 +869,12 @@ bgpq_expand_irrd(struct bgpq_expander *b,
 
 	SX_DEBUG(debug_expander, "expander sending: %s", request);
 
-	if ((ret = write(b->fd, request, strlen(request)) == 0) || ret == -1)
-		err(1, "write");
+	if ((ret = write(b->fd, request, strlen(request)) == 0) || ret == -1) {
+		sx_report(SX_ERROR,
+			"Partial write of request to IRRd: %li bytes, %s\n",
+			ret, strerror(errno));
+		exit(1);
+	}
 
 	memset(response, 0, sizeof(response));
 
@@ -743,7 +886,6 @@ repeat:
 		exit(1);
 	} else if (ret == 0) {
 		sx_report(SX_FATAL, "EOF reading IRRd\n");
-		exit(1);
 	}
 
 	off += ret;
@@ -808,10 +950,8 @@ reread2:
 		if (ret < 0) {
 			sx_report(SX_FATAL, "error reading IRRd: %s\n",
 			    strerror(errno));
-			exit(1);
 		} else if (ret == 0) {
 			sx_report(SX_FATAL, "eof reading IRRd\n");
-			exit(1);
 		}
 		off += ret;
 
@@ -831,7 +971,7 @@ have3:
 			if (c[0] == 0)
 				break;
 			if (callback)
-				callback(c, b, req);
+				if (!callback(c, b, req)) rval = 0;
 			c += spn + 1;
 		}
 		memset(recvbuffer, 0, togot + 2);
@@ -841,20 +981,24 @@ have3:
 		if (b->validate_asns)
 			bgpq_expander_invalidate_asn(b, request);
 	} else if (response[0] == 'D') {
-		/* ... */
+		sx_report(SX_ERROR, "Key not found expanding %s",
+			req->request);
 		if (b->validate_asns)
 			bgpq_expander_invalidate_asn(b, request);
+		rval = 0;
 	} else if (response[0] == 'E') {
 		/* XXXXXX */
 	} else if (response[0] == 'F') {
-		/* XXXXXX */
+		sx_report(SX_ERROR, "Error expanding %s: %s",
+			request, response);
+		rval = 0;
 	} else {
-		sx_report(SX_ERROR,"Wrong reply: %s\n", response);
-		exit(0);
+		sx_report(SX_ERROR,"Wrong reply: %s", response);
+		exit(1);
 	}
 	request_free(req);
 
-	return 0;
+	return rval;
 }
 
 int
@@ -931,6 +1075,7 @@ bgpq_expand(struct bgpq_expander *b)
 	if ((ret = write(fd, "!!\n", 3)) != 3) {
 		sx_report(SX_ERROR, "Partial write of multiple command mode "
 		    "to IRRd: %i bytes, %s\n", ret, strerror(errno));
+		close(fd);
 		exit(1);
 	}
 
@@ -944,6 +1089,7 @@ bgpq_expand(struct bgpq_expander *b)
 				sx_report(SX_ERROR, "Partial write of "
 				    "identifier to IRRd: %i bytes, %s\n",
 				    ret, strerror(errno));
+				close(fd);
 				exit(1);
 			}
 			memset(ident, 0, sizeof(ident));
@@ -951,10 +1097,12 @@ bgpq_expand(struct bgpq_expander *b)
 				SX_DEBUG(debug_expander, "Got answer %s", ident);
 			} else {
 				sx_report(SX_ERROR, "ident, failed read from IRRd\n");
+				close(fd);
 				exit(1);
 			}
 		} else {
 			sx_report(SX_ERROR, "snprintf(ident) failed\n");
+			close(fd);
 			exit(1);
 		}
 	}
@@ -967,6 +1115,7 @@ bgpq_expand(struct bgpq_expander *b)
 		if ((ret = write(fd, "!a\n", 3)) != 3) {
 			sx_report(SX_ERROR, "Partial write of '!a' test query "
 			    "to IRRd: %i bytes, %s\n", ret, strerror(errno));
+			close(fd);
 			exit(1);
 		}
 		memset(aret, 0, sizeof(aret));
@@ -979,14 +1128,26 @@ bgpq_expand(struct bgpq_expander *b)
 			}
 		} else {
 			sx_report(SX_ERROR, "A query test failed read from IRRd\n");
+			close(fd);
 			exit(1);
 		}
 	}
 
+	if (b->usesource) {
+		if (b->sources && b->sources[0] != 0) {
+			b->defaultsources = (char*)calloc(1, strlen(b->sources));
+			strcpy(b->defaultsources, b->sources);
+		} else {
+			b->defaultsources = bgpq_get_irrd_sources(b->fd);
+		}
+	} else {
+		b->defaultsources = bgpq_get_irrd_sources(b->fd);
+	}
+
 	if (b->sources && b->sources[0] != 0) {
 		int slen = strlen(b->sources) + 4;
-		if (slen < 128)
-			slen = 128;
+		if (slen < 256)
+			slen = 256;
 		char sources[slen];
 		slen = snprintf(sources, sizeof(sources), "!s%s\n", b->sources);
 		if (slen > 0) {
@@ -994,6 +1155,7 @@ bgpq_expand(struct bgpq_expander *b)
 			if ((ret = write(fd, sources, slen)) != slen) {
 				sx_report(SX_ERROR, "Partial write of sources to "
 				    "IRRd: %i bytes, %s\n", ret, strerror(errno));
+				close(fd);
 				exit(1);
 			}
 			memset(sources, 0, sizeof(sources));
@@ -1002,14 +1164,17 @@ bgpq_expand(struct bgpq_expander *b)
 				if (sources[0] != 'C') {
 					sx_report(SX_ERROR, "Invalid source(s) "
 					    "'%s': %s\n", b->sources, sources);
+					close(fd);
 					exit(1);
 				}
 			} else {
 				sx_report(SX_ERROR, "failed to read sources\n");
+				close(fd);
 				exit(1);
 			}
 		} else {
 			sx_report(SX_ERROR, "snprintf(sources) failed\n");
+			close(fd);
 			exit(1);
 		}
 	}
@@ -1019,25 +1184,57 @@ bgpq_expand(struct bgpq_expander *b)
 
 	STAILQ_FOREACH(mc, &b->macroses, entry) {
 		if (!b->maxdepth && RB_EMPTY(&b->stoplist)) {
-			if (aquery)
+			if (b->usesource) {
+				char *source = bgpq_get_source(mc->text);
+				if (source){
+					if (pipelining){
+						bgpq_pipeline(b, NULL, NULL, "!s%s\n", source);
+						bgpq_pipeline(b, bgpq_expanded_macro_limit, b,
+							"!i%s\n", bgpq_get_asset(mc->text));
+					} else {
+						bgpq_expand_irrd(b, NULL, NULL, "!s%s\n", source);
+						bgpq_expand_irrd(b, bgpq_expanded_macro_limit,
+							b, "!i%s\n", bgpq_get_asset(mc->text));
+					}
+					free(source);
+				} else {
+					if (pipelining){
+						bgpq_pipeline(b, NULL, NULL, "!s%s\n",
+							b->defaultsources);
+						bgpq_pipeline(b, bgpq_expanded_macro_limit, b,
+							"!i%s\n", bgpq_get_asset(mc->text));
+					} else {
+						bgpq_expand_irrd(b, NULL, NULL, "!s%s\n",
+							b->defaultsources);
+						bgpq_expand_irrd(b, bgpq_expanded_macro_limit, b,
+							"!i%s\n", bgpq_get_asset(mc->text));
+					}
+				}
+			} else if (aquery)
 				bgpq_expand_irrd(b, bgpq_expanded_prefix, b,
 				    "!a%s%s\n",
 				    b->family == AF_INET ? "4" : "6",
-				    mc->text);
+				    bgpq_get_asset(mc->text));
 			else
 				bgpq_expand_irrd(b, bgpq_expanded_macro, b,
-				    "!i%s,1\n", mc->text);
+				    "!i%s,1\n", bgpq_get_asset(mc->text));
 		} else {
-			bgpq_expander_add_already(b, mc->text);
+			bgpq_expander_add_already(b, bgpq_get_asset(mc->text));
 			if (pipelining)
 				bgpq_pipeline(b, bgpq_expanded_macro_limit,
-				    NULL, "!i%s\n", mc->text);
+				    NULL, "!i%s\n", bgpq_get_asset(mc->text));
 			else
 				bgpq_expand_irrd(b, bgpq_expanded_macro_limit,
-				    NULL, "!i%s\n", mc->text);
+				    NULL, "!i%s\n", bgpq_get_asset(mc->text));
 		}
 	}
 
+	if (pipelining){
+		bgpq_pipeline(b, NULL, NULL, "!s%s\n", b->defaultsources);
+	} else {
+		bgpq_expand_irrd(b, NULL, NULL, "!s%s\n", b->defaultsources);
+	}
+
 	if (pipelining) {
 		if (!STAILQ_EMPTY(&b->wq))
 			bgpq_write(b);
@@ -1047,12 +1244,70 @@ bgpq_expand(struct bgpq_expander *b)
 
 	if (b->generation >= T_PREFIXLIST || b->validate_asns) {
 		STAILQ_FOREACH(mc, &b->rsets, entry) {
-			if (b->family == AF_INET)
-				bgpq_expand_irrd(b, bgpq_expanded_prefix,
-				    NULL, "!i%s,1\n", mc->text);
-			else
-				bgpq_expand_irrd(b, bgpq_expanded_v6prefix,
-				    NULL, "!i%s,1\n", mc->text);
+			if (b->usesource) {
+				char *source = bgpq_get_source(mc->text);
+				if (source){
+					if (pipelining){
+						printf("Checking %s\n", bgpq_get_rset(mc->text));
+						bgpq_pipeline(b, NULL, NULL, "!s%s\n", source);
+						if (b->family == AF_INET)
+							bgpq_pipeline(b, bgpq_expanded_prefix,
+				    			NULL, "!i%s\n", bgpq_get_rset(mc->text));
+						else
+							bgpq_pipeline(b, bgpq_expanded_v6prefix,
+								NULL, "!i%s\n", bgpq_get_rset(mc->text));
+					} else {
+						bgpq_expand_irrd(b, NULL, NULL, "!s%s\n", source);
+						if (b->family == AF_INET)
+							bgpq_expand_irrd(b, bgpq_expanded_prefix,
+				    			NULL, "!i%s\n", bgpq_get_rset(mc->text));
+						else
+							bgpq_expand_irrd(b, bgpq_expanded_v6prefix,
+								NULL, "!i%s\n", bgpq_get_rset(mc->text));
+					}
+					free(source);
+				} else {
+					if (pipelining){
+						bgpq_pipeline(b, NULL, NULL, "!s%s\n",
+							b->defaultsources);
+						if (b->family == AF_INET)
+							bgpq_pipeline(b, bgpq_expanded_prefix,
+								NULL, "!i%s\n", bgpq_get_rset(mc->text));
+						else
+							bgpq_pipeline(b, bgpq_expanded_v6prefix,
+								NULL, "!i%s\n", bgpq_get_rset(mc->text));
+					} else {
+						bgpq_expand_irrd(b, NULL, NULL, "!s%s\n",
+							b->defaultsources);
+						if (b->family == AF_INET)
+							bgpq_expand_irrd(b, bgpq_expanded_prefix,
+								NULL, "!i%s\n", bgpq_get_rset(mc->text));
+						else
+							bgpq_expand_irrd(b, bgpq_expanded_v6prefix,
+								NULL, "!i%s\n", bgpq_get_rset(mc->text));
+					}
+				}
+			} else {
+				if (pipelining){
+					bgpq_pipeline(b, NULL, NULL, "!s%s\n",
+						b->defaultsources);
+					if (b->family == AF_INET)
+						bgpq_pipeline(b, bgpq_expanded_prefix,
+							NULL, "!i%s,1\n", bgpq_get_rset(mc->text));
+					else
+						bgpq_pipeline(b, bgpq_expanded_v6prefix,
+							NULL, "!i%s,1\n", bgpq_get_rset(mc->text));
+				} else {
+					bgpq_expand_irrd(b, NULL, NULL, "!s%s\n",
+						b->defaultsources);
+					if (b->family == AF_INET)
+						bgpq_expand_irrd(b, bgpq_expanded_prefix,
+							NULL, "!i%s,1\n", bgpq_get_rset(mc->text));
+					else
+						bgpq_expand_irrd(b, bgpq_expanded_v6prefix,
+							NULL, "!i%s,1\n", bgpq_get_rset(mc->text));
+				}
+			}
 		}
 
 		RB_FOREACH(asne, asn_tree, &b->asnlist) {
@@ -1094,6 +1349,7 @@ bgpq_expand(struct bgpq_expander *b)
 		fcntl(fd, F_SETFL, fl);
 	}
 	close(fd);
+	free(b->defaultsources);
 
 	return 1;
 }
@@ -1113,7 +1369,7 @@ sx_radix_node_freeall(struct sx_radix_node *n) {
 	if (n->payload)
 		free(n->payload);
 
-	sx_prefix_destroy(n->prefix);
+	sx_prefix_free(n->prefix);
 
 	free(n);
 }

extern.h

@@ -91,6 +91,8 @@ struct bgpq_expander {
 	struct sx_radix_tree	 	*tree;
 	int			 	 family;
 	char				*sources;
+	char				*defaultsources;
+	unsigned int			 usesource;
 	uint32_t		 	 asnumber;
 	int			 	 aswidth;
 	char				*name;
@@ -126,6 +128,10 @@ int bgpq_expander_add_prefix(struct bgpq_expander *b, char *prefix);
 int bgpq_expander_add_prefix_range(struct bgpq_expander *b, char *prefix);
 int bgpq_expander_add_stop(struct bgpq_expander *b, char *object);
 
+char* bgpq_get_asset(char *object);
+char* bgpq_get_rset(char *object);
+char* bgpq_get_source(char *object);
+
 int bgpq_expand(struct bgpq_expander *b);
 
 void bgpq4_print_prefixlist(FILE *f, struct bgpq_expander *b);

main.c

@@ -190,7 +190,7 @@ main(int argc, char* argv[])
 		exit(1);
 	}
 #endif
-	
+
 	bgpq_expander_init(&expander, af);
 
 	if (getenv("IRRD_SOURCES"))
@@ -694,17 +694,23 @@ main(int argc, char* argv[])
 		usage(1);
 
 	while (argv[0]) {
+		char *obj = argv[0];
+		char *delim = strstr(argv[0], "::");
+		if (delim) {
+			expander.usesource = 1;
+			obj = delim + 2;
+		}
 		if (!strcmp(argv[0], "EXCEPT")) {
 			exceptmode = 1;
 		} else if (exceptmode) {
 			bgpq_expander_add_stop(&expander, argv[0]);
-		} else if (!strncasecmp(argv[0], "AS-", 3)) {
+		} else if (!strncasecmp(obj, "AS-", 3)) {
 			bgpq_expander_add_asset(&expander, argv[0]);
-		} else if (!strncasecmp(argv[0], "RS-", 3)) {
+		} else if (!strncasecmp(obj, "RS-", 3)) {
 			bgpq_expander_add_rset(&expander, argv[0]);
-		} else if (!strncasecmp(argv[0], "AS", 2)) {
+		} else if (!strncasecmp(obj, "AS", 2)) {
 			char *ec;
-			if ((ec = strchr(argv[0], ':'))) {
+			if ((ec = strchr(obj, ':'))) {
 				if (!strncasecmp(ec + 1, "AS-", 3)) {
 					bgpq_expander_add_asset(&expander, argv[0]);
 				} else if (!strncasecmp(ec + 1, "RS-", 3)) {

sx_prefix.c

@@ -26,6 +26,7 @@
  */
 
 #include <ctype.h>
+#include <err.h>
 #include <errno.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -38,13 +39,13 @@
 int debug_aggregation = 0;
 extern int debug_expander;
 
-struct sx_prefix*
+struct sx_prefix *
 sx_prefix_alloc(struct sx_prefix *p)
 {
-	struct sx_prefix *sp = malloc(sizeof(struct sx_prefix));
+	struct sx_prefix *sp;
 
-	if (!sp)
-		return NULL;
+	if ((sp = malloc(sizeof(struct sx_prefix))) == NULL)
+		err(1, NULL);
 
 	if (p)
 		memcpy(sp, p, sizeof(struct sx_prefix));
@@ -55,7 +56,7 @@ sx_prefix_alloc(struct sx_prefix *p)
 }
 
 void
-sx_prefix_destroy(struct sx_prefix *p)
+sx_prefix_free(struct sx_prefix *p)
 {
 	if (p)
 		free(p);
@@ -64,15 +65,16 @@ sx_prefix_destroy(struct sx_prefix *p)
 void
 sx_radix_node_destroy(struct sx_radix_node *n)
 {
-	if (n) {
-		if (n->payload)
-			free(n->payload);
+	if (!n)
+		return;
 
-    		if (n->prefix)
-			free(n->prefix);
+	if (n->payload)
+		free(n->payload);
 
-    		free(n);
-	}
+	if (n->prefix)
+		free(n->prefix);
+
+	free(n);
 }
 
 void
@@ -140,7 +142,7 @@ sx_prefix_parse(struct sx_prefix *p, int af, char *text)
 	c = strchr(mtext,'/');
 
 	if (c) {
-		char* eod;
+		char *eod;
 		*c = 0;
 		masklen = strtol(c + 1, &eod, 10);
 		if (eod && eod[0] && !isspace(eod[0])) {
@@ -257,25 +259,19 @@ sx_prefix_setbit(struct sx_prefix *p, int n)
 
 
 static int
-sx_radix_tree_insert_specifics(struct sx_radix_tree *t, struct sx_prefix *p,
+sx_radix_tree_insert_specifics(struct sx_radix_tree *t, struct sx_prefix p,
     unsigned min, unsigned max)
 {
-	struct sx_prefix *np;
-	np = sx_prefix_alloc(p);
+	if (p.masklen >= min)
+		sx_radix_tree_insert(t, &p);
 
-	if (np->masklen >= min) {
-		struct sx_radix_node *nn = sx_radix_tree_insert(t, np);
-		sx_prefix_destroy(np);
-		np = nn->prefix;
-	}
-
-	if (np->masklen + 1 > max)
+	if (p.masklen + 1 > max)
 		return 1;
 
-	np->masklen += 1;
-	sx_radix_tree_insert_specifics(t, np, min, max);
-	sx_prefix_setbit(np, np->masklen);
-	sx_radix_tree_insert_specifics(t, np, min, max);
+	p.masklen += 1;
+	sx_radix_tree_insert_specifics(t, p, min, max);
+	sx_prefix_setbit(&p, p.masklen);
+	sx_radix_tree_insert_specifics(t, p, min, max);
 
 	return 1;
 }
@@ -284,18 +280,16 @@ int
 sx_prefix_range_parse(struct sx_radix_tree *tree, int af, unsigned int maxlen,
     char *text)
 {
-	char			*d = strchr(text, '^');
-	struct sx_prefix	*p;
+	struct sx_prefix	 p;
 	unsigned long		 min, max = 0;
-
-	p = sx_prefix_alloc(NULL);
+	char			*d = strchr(text, '^');
 
 	if (!d || !d[1])
 		return 0;
 
 	*d = 0;
 
-	if (!sx_prefix_parse(p, 0, text)) {
+	if (!sx_prefix_parse(&p, 0, text)) {
 		sx_report(SX_ERROR, "Unable to parse prefix %s^%s\n", text,
 		    d + 1);
 		return 0;
@@ -303,23 +297,23 @@ sx_prefix_range_parse(struct sx_radix_tree *tree, int af, unsigned int maxlen,
 
 	*d = '^';
 
-	if (af && p->family != af) {
+	if (af && p.family != af) {
 		sx_report(SX_ERROR, "Ignoring prefix %s, wrong af %i\n", text,
-		    p->family);
+		    p.family);
 		return 0;
 	}
 
-	if (maxlen && p->masklen > maxlen) {
+	if (maxlen && p.masklen > maxlen) {
 		SX_DEBUG(debug_expander, "Ignoring prefix %s, masklen %i > max"
-		    " masklen %u\n", text, p->masklen, maxlen);
+		    " masklen %u\n", text, p.masklen, maxlen);
 		return 0;
 	}
 
 	if (d[1] == '-') {
-		min = p->masklen + 1;
+		min = p.masklen + 1;
 		max = maxlen;
 	} else if (d[1] == '+') {
-		min = p->masklen;
+		min = p.masklen;
 		max = maxlen;
 	} else if (isdigit(d[1])) {
 		char *dm = NULL;
@@ -336,9 +330,9 @@ sx_prefix_range_parse(struct sx_radix_tree *tree, int af, unsigned int maxlen,
 		return 0;
 	}
 
-	if (min < p->masklen) {
+	if (min < p.masklen) {
 		sx_report(SX_ERROR, "Invalid prefix-range %s: min %lu < "
-		    "masklen %u\n", text, min, p->masklen);
+		    "masklen %u\n", text, min, p.masklen);
 		return 0;
 	}
 
@@ -377,7 +371,7 @@ sx_prefix_new(int af, char *text)
 		return NULL;
 
 	if (!sx_prefix_parse(p, af, text)) {
-		sx_prefix_destroy(p);
+		sx_prefix_free(p);
 		return NULL;
 	}
 
@@ -519,13 +513,13 @@ sx_prefix_jsnprintf(struct sx_prefix *p, char *rbuffer, int srb)
 	return snprintf(rbuffer, srb, "%s\\/%i", buffer, p->masklen);
 }
 
-struct sx_radix_tree*
+struct sx_radix_tree *
 sx_radix_tree_new(int af)
 {
-	struct sx_radix_tree *rt = malloc(sizeof(struct sx_radix_tree));
+	struct sx_radix_tree *rt;
 
-	if (!rt)
-		return NULL;
+	if ((rt = malloc(sizeof(struct sx_radix_tree))) == NULL)
+		err(1, NULL);
 
 	memset(rt, 0, sizeof(struct sx_radix_tree));
 	rt->family = af;
@@ -539,13 +533,13 @@ sx_radix_tree_empty(struct sx_radix_tree *t)
 	return t->head == NULL;
 }
 
-struct sx_radix_node*
+struct sx_radix_node *
 sx_radix_node_new(struct sx_prefix *prefix)
 {
-	struct sx_radix_node *rn = malloc(sizeof(struct sx_radix_node));
+	struct sx_radix_node *rn;
 
-	if (!rn)
-		return NULL;
+	if ((rn = malloc(sizeof(struct sx_radix_node))) == NULL)
+		err(1, NULL);
 
 	memset(rn, 0, sizeof(struct sx_radix_node));
 
@@ -582,7 +576,7 @@ sx_prefix_eqbits(struct sx_prefix *a, struct sx_prefix *b)
 	return b->masklen;
 }
 
-struct sx_prefix*
+struct sx_prefix *
 sx_prefix_overlay(struct sx_prefix *p, int n)
 {
 	struct sx_prefix *sp = sx_prefix_alloc(p);
@@ -668,7 +662,7 @@ next:
 }
 	
 			
-struct sx_radix_node*
+struct sx_radix_node *
 sx_radix_tree_lookup(struct sx_radix_tree *tree, struct sx_prefix *prefix)
 {
 	unsigned int 		 eb;
@@ -725,18 +719,18 @@ next:
 		char pbuffer[128], cbuffer[128];
 		sx_prefix_snprintf(prefix, pbuffer, sizeof(pbuffer));
 		sx_prefix_snprintf(chead->prefix, cbuffer, sizeof(cbuffer));
-		printf("Unreachible point... eb=%i, prefix=%s, chead=%s\n",
+		printf("Unreachable point... eb=%i, prefix=%s, chead=%s\n",
 		    eb, pbuffer, cbuffer);
 		abort();
 	}
 }
 
 
-struct sx_radix_node*
+struct sx_radix_node *
 sx_radix_tree_insert(struct sx_radix_tree *tree, struct sx_prefix *prefix)
 {
-	unsigned int		 eb;
-	struct sx_radix_node	**candidate=NULL, *chead;
+	unsigned int eb;
+	struct sx_radix_node *chead, **candidate = NULL;
 
 	if (!tree || !prefix)
 		return NULL;
@@ -752,7 +746,7 @@ sx_radix_tree_insert(struct sx_radix_tree *tree, struct sx_prefix *prefix)
 	candidate = &tree->head;
 	chead = tree->head;
 
-next:
+ next:
 	eb = sx_prefix_eqbits(prefix, chead->prefix);
 	if (eb < prefix->masklen && eb < chead->prefix->masklen) {
 		struct sx_prefix *neoRoot = sx_prefix_alloc(prefix);
@@ -761,7 +755,7 @@ next:
 		neoRoot->masklen = eb;
 		sx_prefix_adjust_masklen(neoRoot);
 		rn=sx_radix_node_new(neoRoot);
-                sx_prefix_destroy(neoRoot);
+                sx_prefix_free(neoRoot);
                 neoRoot = rn->prefix;
 		if (!rn) {
 			sx_report(SX_ERROR,"Unable to create node: %s\n",
@@ -785,11 +779,10 @@ next:
 		return ret;
 	} else if (eb == prefix->masklen && eb < chead->prefix->masklen) {
 		struct sx_radix_node *ret = sx_radix_node_new(prefix);
-		if (sx_prefix_isbitset(chead->prefix, eb + 1)) {
+		if (sx_prefix_isbitset(chead->prefix, eb + 1))
 			ret->r = chead;
-		} else {
+		else
 			ret->l = chead;
-		}
 		ret->parent = chead->parent;
 		chead->parent = ret;
 		*candidate = ret;
@@ -826,7 +819,7 @@ next:
 		char pbuffer[128], cbuffer[128];
 		sx_prefix_snprintf(prefix, pbuffer, sizeof(pbuffer));
 		sx_prefix_snprintf(chead->prefix, cbuffer, sizeof(cbuffer));
-		printf("Unreachible point... eb=%i, prefix=%s, chead=%s\n", eb,
+		printf("Unreachable point... eb=%i, prefix=%s, chead=%s\n", eb,
 		    pbuffer, cbuffer);
 		abort();
 	}
@@ -848,7 +841,7 @@ sx_radix_node_fprintf(struct sx_radix_node *node, void *udata)
 
 int
 sx_radix_node_foreach(struct sx_radix_node *node,
-    void (*func)(struct sx_radix_node*, void*), void *udata)
+    void (*func)(struct sx_radix_node *, void *), void *udata)
 {
 	func(node, udata);
 
@@ -1055,7 +1048,7 @@ sx_radix_tree_aggregate(struct sx_radix_tree *tree)
 static void
 setGlueUpTo(struct sx_radix_node *node, void *udata)
 {
-	unsigned refine = *(unsigned*)udata;
+	unsigned refine = *(unsigned *)udata;
 
 	if (node && node->prefix->masklen <= refine)
 		node->isGlue = 1;
@@ -1115,7 +1108,7 @@ sx_radix_tree_refine(struct sx_radix_tree *tree, unsigned refine)
 static void
 setGlueFrom(struct sx_radix_node *node, void *udata)
 {
-	unsigned refine = *(unsigned*)udata;
+	unsigned refine = *(unsigned *)udata;
 
 	if (node && node->prefix->masklen <= refine)
 		node->isGlue = 1;

sx_prefix.h

@@ -68,7 +68,7 @@ struct sx_radix_node *sx_radix_tree_lookup_exact(struct sx_radix_tree *tree,
 	struct sx_prefix *prefix);
 
 struct sx_prefix *sx_prefix_alloc(struct sx_prefix *p);
-void sx_prefix_destroy(struct sx_prefix *p);
+void sx_prefix_free(struct sx_prefix *p);
 void sx_radix_node_destroy(struct sx_radix_node *p);
 void sx_prefix_adjust_masklen(struct sx_prefix *p);
 struct sx_prefix *sx_prefix_new(int af, char *text);