huwcampbell.com
❄️ The Icicle Streaming Query Language ❄️
1Outputting Java Bytecode
2========================
3
4We want to convert flattened Avalanche to jbc.
5It looks like Jacob Stanley has the majority of a decent library for doing this: [https://github.com/jystic/tonic].
6
7There are other libraries which may be better baked, for example hs-java, but I just dislike its imperative way of constructing bytecode.
8I think I will start with Jacob's library, and if I run into too many shortcomings, switch to hs-java.
9
10IcicleState
11==========
12```
13// The type parameter here is the type of the concrete feature
14// We do not have a type parameter for the virtual feature, because
15// we may have multiple virtual features being computed.
16interface IcicleState<T>
17{
18 // There is a phantom field keeping track of allowed operations in a given "state".
19 // This would be tracked by types normally, but keeping everything in one place
20 // actually makes code generation easier.
21
22 // However for sanity, I need to keep track of this stuff, so I will use pre and postconditions.
23
24 // data State = Start | ReadingHistory | ReadingNew | Done
25 // state : State = Start
26
27 // Class-wide invariant:
28 // state can only "increase": Start -> Done, Start -> ReadingNew, but not ReadingHistory -> Start.
29 // state <= state'
30
31 // We also have extra phantom fields for keeping track of whether a value can be read:
32 // canPull : Bool = False
33 // hasRow : Bool = False
34
35
36 // Get the date for which we are calculating the feature.
37 //
38 // Pre: state == Start
39 //
40 // Post: -
41 DateTime snapshotDate();
42
43
44 // For resumable accumulators like "sum over all time",
45 // load the last saved value.
46 //
47 // Pre: state == Start
48 // /\ virtualFeature in features
49 //
50 // Post: null if no value, or last saved value for (virtualFeature, accumulatorName).
51 U loadResumable<U>(String virtualFeature, String accumulatorName);
52
53
54 // For resumable accumulators like "sum over all time",
55 // load the last saved value.
56 //
57 // Pre: state == Done
58 // /\ virtualFeature in features
59 // /\ value != null
60 //
61 // Post: -
62 void saveResumable<U>(String virtualFeature, String accumulatorName, U value);
63
64
65 // Record actual output value of computed virtual feature
66 //
67 // Pre: state == Done
68 // /\ virtualFeature in features
69 // /\ value != null
70 //
71 // Post: -
72 void output<U>(String virtualFeature, U value);
73
74
75 // Start reading history
76 //
77 // Pre: state == Start
78 // /\ ! hasRow
79 // /\ ! canPull
80 //
81 // Post: state' == ReadingHistory
82 // /\ ! hasRow'
83 // /\ canPull'
84 void startHistory();
85
86
87 // Start reading new stuff
88 //
89 // Pre: (state == Start \/ state == ReadingHistory)
90 // /\ ! hasRow
91 // /\ ! canPull
92 //
93 // Post: state' == ReadingNew
94 // /\ ! hasRow'
95 // /\ canPull'
96 void startNew();
97
98
99 // Pull the next row in
100 //
101 // Pre: (state == ReadingHistory \/ state == ReadingNew)
102 // /\ canPull
103 //
104 // Post: hasRow' == canPull' == return
105 // at end of ReadingNew, state' becomes Done
106 boolean nextRow();
107
108
109 // Note that the current fact is used
110 //
111 // Pre: (state == ReadingHistory \/ state == ReadingNew)
112 // /\ hasRow
113 //
114 // Post: on the next run, this fact will be included in startHistory / nextRow
115 void keepFactInHistory();
116
117
118 // Read the value of the current fact
119 //
120 // Pre: (state == ReadingHistory \/ state == ReadingNew)
121 // /\ hasRow
122 //
123 // Post: -
124 T currentRow();
125
126}
127```
128
129Examples
130========
131
132CountAboveTen
133-------------
134
135Let's start with a very simple one.
136Given the input query:
137```
138feature salary ~> filter value > 10 ~> count
139```
140
141We get the following flattened Avalanche.
142```
143gen$date = DATE
144{
145 init acc$conv$5 = 0 : Int (Resumable);
146 for_facts (gen$fact : Int) in new {
147 let anf$3 = fst# [Int] [DateTime] gen$fact;
148 if (gt# [Int] anf$3 (10 : Int)) {
149 read acc$conv$5 = acc$conv$5;
150 write acc$conv$5 = add# acc$conv$5 (1 : Int);
151 }
152 }
153 read conv$5 = acc$conv$5;
154 return conv$5;
155}
156```
157
158I'm imagining the Java would be something like this.
159```
160void compute(IcicleState<Integer> icicle)
161{
162 // gen$date = DATE
163 DateTime gen_date = icicle.snapshotDate();
164
165 // init acc$conv$5 = 0 : Int (Resumable);
166 Integer acc_conv_5_load = icicle.loadResumable<Integer>("CountAboveTen", "acc$conv$5");
167 int acc_conv_5 = 0;
168 if (acc_conv_5_load != null) {
169 acc_conv_5 = acc_conv_5_load.value();
170 }
171
172 // for_facts (gen$fact : Int) in new
173 icicle.startNew();
174 while (icicle.nextRow()) {
175 // let anf$3 = fst# [Int] [DateTime] gen$fact;
176 int anf_3 = icicle.currentRow().value();
177
178 // if (gt# [Int] anf$3 (10 : Int))
179 if (anf_3 > 10) {
180 // read acc$conv$5 = acc$conv$5;
181 int read_acc_conv_5 = acc_conv_5;
182 // write acc$conv$5 = add# acc$conv$5 (1 : Int);
183 acc_conv_5 = read_acc_conv_5 + 1;
184 }
185 }
186
187 // init acc$conv$5 = 0 : Int (Resumable);
188 icicle.saveResumable<Integer>("CountAboveTen", "acc$conv$5", new Integer(acc_conv_5));
189
190 // read conv$5 = acc$conv$5;
191 int conv_5 = acc_conv_5
192 // return conv$5;
193 icicle.output<Integer>("CountAboveTen", new Integer(conv_5));
194}
195```
196
197And the very very rough bytecode.
198This is not quite right syntax but the idea is there.
199It could also be improved a bit like not storing variables before using them just once.
200```
201// void Compute(IcicleState icicle)
202 aload ICICLE
203 invokeinterface #IcicleState.snapShotDate
204 astore GEN_DATE // gen_date = icicle.snapShotDate()
205
206
207 aload ICICLE
208 ldc "CountAboveTen"
209 ldc "acc$conv$5"
210 invokeinterface #IcicleState.loadResumable
211 astore ACC_CONV_5_LOAD // acc_conv_5_load = icicle.loadResumable(...)
212
213
214 iconst 0
215 istore ACC_CONV_5 // acc_conv_5 = 0
216
217
218 aload ACC_CONV_5_LOAD // acc_conv_5_load
219 aconstnull // null
220 if_acmpne BR_LOAD_ACC_CONV_5 // if (acc_conv_5_load != null)
221 goto BR_SKIP_ACC_CONV_5
222
223BR_LOAD_ACC_CONV_5:
224
225 aload ACC_CONV_5_LOAD
226 invokestatic #Integer.value
227 istore ACC_CONV_5 // acc_conv_5 = acc_conv_5_load.value()
228
229BR_SKIP_ACC_CONV_5:
230
231 aload ICICLE
232 invokevirtual #IcicleState.startNew
233
234BR_WHILE_START:
235
236 aload ICICLE
237 invokevirtual #IcicleState.nextRow
238 ifeq BR_WHILE_END // if (icicle.nextRow() == 0) goto end
239
240
241 aload ICICLE
242 invokevirtual #IcicleState.currentRow
243 checkcast #Integer
244 invokevirtual #Integer.value
245 istore ANF_3 // anf_3 = icicle.currentRow().value();
246
247 iload ANF_3
248 iconst 10
249 if_icmpgt BR_ANF_3_THEN // if (anf_3 > 10)
250 goto BR_ANF_3_ELSE
251
252BR_ANF_3_THEN:
253 iload ACC_CONV_5
254 istore READ_ACC_CONV_5 // read_acc_conv_5 = acc_conv_5;
255
256 iload READ_ACC_CONV_5
257 iconst 1
258 iadd
259 istore ACC_CONV_5 // acc_conv_5 = read_acc_conv_5 + 1
260
261BR_ANF_3_ELSE:
262 goto BR_WHILE_START
263
264
265BR_WHILE_END:
266
267 aload ICICLE
268 ldc "CountAboveTen"
269 ldc "acc$conv$5"
270
271 new #Integer
272 dup
273 iload ACC_CONV_5
274 invokespecial #Integer.<init>
275
276 invokeinterface #IcicleState.saveResumable // icicle.saveResumable<Integer>(...);
277
278 iload ACC_CONV_5
279 istore CONV_5
280
281 aload ICICLE
282 ldc "CountAboveTen"
283
284 new #Integer
285 dup
286 iload ACC_CONV_5
287 invokespecial #Integer.<init>
288
289 invokeinterface #IcicleState.output // icicle.output<Integer>("CountAboveTen", new Integer(conv_5));
290}
291```
292
293
294CountLastWeek
295-------------
296
297How many things happened last week?
298Input query:
299```
300feature salary ~> windowed between 7 days and 14 days ~> count
301```
302
303Flattened Avalanche.
304```
305gen$date = DATE
306{
307 init acc$conv$4 = 0 : Int (Windowed);
308 for_facts (gen$fact : Int) in history {
309 let anf$4 = snd# [Int] [DateTime] gen$fact;
310 let anf$5 = DateTime_daysDifference# anf$4 gen$date;
311 let anf$6 = le# [Int] anf$5 (14 : Int);
312 let anf$8 = ge# [Int] anf$5 (7 : Int);
313 if (and# anf$6 anf$8) {
314 read acc$conv$4 = acc$conv$4;
315 write acc$conv$4 = add# acc$conv$4 (1 : Int);
316 keep_fact_in_history;
317 } else {
318 if (lt# [Int] anf$5 (7 : Int)) {
319 keep_fact_in_history;
320 }
321
322 }
323
324 }
325 for_facts (gen$fact : Int) in new {
326 let anf$16 = snd# [Int] [DateTime] gen$fact;
327 let anf$17 = DateTime_daysDifference# anf$16 gen$date;
328 let anf$18 = le# [Int] anf$17 (14 : Int);
329 let anf$20 = ge# [Int] anf$17 (7 : Int);
330 if (and# anf$18 anf$20) {
331 read acc$conv$4 = acc$conv$4;
332 write acc$conv$4 = add# acc$conv$4 (1 : Int);
333 keep_fact_in_history;
334 } else {
335 if (lt# [Int] anf$17 (7 : Int)) {
336 keep_fact_in_history;
337 }
338
339 }
340
341 }
342 read conv$4 = acc$conv$4;
343 return conv$4;
344}
345```
346
347Java:
348```
349void compute(IcicleState<Integer> icicle)
350{
351 // gen$date = DATE
352 DateTime gen_date = icicle.snapshotDate();
353
354 // init acc$conv$4 = 0 : Int (Windowed);
355 int acc_conv_4 = 0;
356
357 // for_facts (gen$fact : Int) in history
358 icicle.startHistory();
359 while (icicle.nextRow()) {
360 // let anf$4 = snd# [Int] [DateTime] gen$fact;
361 DateTime anf_4 = icicle.currentTime();
362 // let anf$5 = DateTime_daysDifference# anf$4 gen$date;
363 int anf_5 = DateTime.daysDifference(anf_4, gen_date);
364
365 // let anf$6 = le# [Int] anf$5 (14 : Int);
366 bool anf_6 = anf_5 <= 14;
367 // let anf$8 = ge# [Int] anf$5 (7 : Int);
368 bool anf_8 = anf_8 >= 7;
369
370 // if (and# anf$6 anf$8)
371 if (anf_6 && anf_8) {
372 // read acc$conv$4 = acc$conv$4;
373 int acc_conv_4_read = acc_conv_4;
374 // write acc$conv$4 = add# acc$conv$4 (1 : Int);
375 acc_conv_4 = acc_conv_4_read + 1;
376 icicle.keepFactInHistory();
377 } else {
378 // if (lt# [Int] anf$5 (7 : Int))
379 if (anf_5 < 7) {
380 // keep_fact_in_history;
381 icicle.keepFactInHistory();
382 }
383 }
384 }
385
386 // for_facts (gen$fact : Int) in new
387 icicle.startNew();
388 while (icicle.nextRow()) {
389 // let anf$16 = snd# [Int] [DateTime] gen$fact;
390 DateTime anf_16 = icicle.currentTime();
391 // let anf$17 = DateTime_daysDifference# anf$16 gen$date;
392 int anf_17 = DateTime.daysDifference(anf_16, gen_date);
393
394 // let anf$18 = le# [Int] anf$17 (14 : Int);
395 bool anf_18 = anf_17 <= 14;
396 // let anf$20 = ge# [Int] anf$17 (7 : Int);
397 bool anf_20 = anf_17 >= 7;
398
399 // if (and# anf$18 anf$20)
400 if (anf_18 && anf_20) {
401 // read acc$conv$4 = acc$conv$4;
402 int acc_conv_4_read = acc_conv_4;
403 // write acc$conv$4 = add# acc$conv$4 (1 : Int);
404 acc_conv_4 = acc_conv_4_read + 1;
405 icicle.keepFactInHistory();
406 } else {
407 // if (lt# [Int] anf$17 (7 : Int))
408 if (anf_17 < 7) {
409 icicle.keepFactInHistory();
410 }
411 }
412 }
413
414 // read conv$4 = acc$conv$4;
415 int conv_4 = acc_conv_4
416 // return conv$4;
417 icicle.output<Integer>("CountLastWeek", new Integer(conv_4));
418}
419```
420