You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 00:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Jul 12, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
307982235167,22sleep20-21swapper/207:06:112
307712232161,58sleep00-21swapper/007:05:490
308042202167,23sleep30-21swapper/307:06:163
307772201164,24sleep10-21swapper/107:05:541
1383721820,4sleep03117999cyclictest07:40:200
31192998884,3cyclictest0-21swapper/209:49:152
57892530,3sleep339-21rcuc/308:35:213
31192995248,3cyclictest0-21swapper/207:57:352
176902500,2sleep30-21swapper/311:30:183
27882490,1sleep02765-21gltestperf12:10:160
143462460,2sleep20-21swapper/207:40:262
3117999441,31cyclictest20716-21grep11:35:260
3117999401,30cyclictest28900-21/usr/sbin/munin09:30:210
31179994011,13cyclictest29903-21munin-run12:00:010
3119699391,28cyclictest0-21swapper/311:10:513
3119699390,32cyclictest0-21swapper/309:59:113
3117999397,20cyclictest10355-21/usr/sbin/munin11:15:270
3117999391,26cyclictest29282-21latency_hist10:45:010
3117999391,24cyclictest25551-21ntpq09:20:220
3117999381,28cyclictest27549-21kworker/u8:212:20:170
3117999381,26cyclictest31484-21latency_hist10:50:010
3117999380,30cyclictest0-21swapper/009:50:190
3117999377,21cyclictest13166-21/usr/sbin/munin12:35:250
3117999376,21cyclictest27070-21fschecks_count09:25:140
3117999371,32cyclictest12707-21grep11:20:130
3117999371,26cyclictest31116-21grep09:35:130
3117999371,26cyclictest31116-21grep09:35:130
3117999371,25cyclictest4114-21/usr/sbin/munin07:20:260
3119299361,11cyclictest0-21swapper/209:19:032
3117999362,22cyclictest87650irq/129-enp1s0-09:15:200
3117999361,26cyclictest25881-21if_enp2s010:35:180
3119699351,26cyclictest0-21swapper/312:23:553
3119699351,24cyclictest0-21swapper/311:37:513
3119299351,9cyclictest0-21swapper/211:42:192
3119299351,9cyclictest0-21swapper/211:22:392
3119299351,9cyclictest0-21swapper/211:07:152
3119299351,11cyclictest0-21swapper/209:50:072
3119299351,10cyclictest0-21swapper/212:20:112
3117999354,16cyclictest87550irq/128-enp1s0-07:10:280
3117999354,16cyclictest87550irq/128-enp1s0-07:10:280
3117999353,28cyclictest31657-21grep12:00:260
3117999353,28cyclictest31657-21grep12:00:260
3117999352,24cyclictest20772-21gltestperf09:10:150
3117999352,23cyclictest8327-21cat07:30:010
3117999351,31cyclictest21143-21fschecks_count10:25:140
3119699340,25cyclictest0-21swapper/310:20:313
3119299341,10cyclictest0-21swapper/209:56:392
3117999347,19cyclictest0-21swapper/008:05:180
3117999345,16cyclictest2078-21latency_hist12:10:010
3117999345,15cyclictest23979-21fschecks_count11:45:150
3117999342,21cyclictest87650irq/129-enp1s0-11:35:020
3117999341,29cyclictest13912-21munin-run10:10:010
3117999340,4cyclictest5492-21kworker/u8:211:10:230
3119699331,4cyclictest9421-21memory07:30:193
3119699331,15cyclictest0-21swapper/307:45:073
3119299331,8cyclictest0-21swapper/210:15:392
3119299331,19cyclictest0-21swapper/212:01:512
3119299331,19cyclictest0-21swapper/212:01:512
3119299331,10cyclictest0-21swapper/210:02:152
3119299330,9cyclictest0-21swapper/207:52:272
3119299330,20cyclictest0-21swapper/211:04:312
31179993312,18cyclictest10015-21sendmail_mailst12:25:240
3117999331,20cyclictest827-21runrttasks10:19:060
3117999331,20cyclictest31717-21/usr/sbin/munin10:50:210
3117999331,20cyclictest31717-21/usr/sbin/munin10:50:210
3117999331,18cyclictest0-21swapper/011:04:400
3119699321,28cyclictest0-21swapper/312:08:203
3119699321,23cyclictest0-21swapper/312:38:273
3119699320,12cyclictest0-21swapper/310:06:033
3119299321,9cyclictest11665-21diskstats08:50:152
3119299321,9cyclictest0-21swapper/209:37:352
3119299321,9cyclictest0-21swapper/209:37:352
3119299321,28cyclictest0-21swapper/211:13:552
3119299321,11cyclictest0-21swapper/210:26:272
3119299321,10cyclictest0-21swapper/212:25:112
3119299321,10cyclictest0-21swapper/209:30:092
3119299320,9cyclictest0-21swapper/209:29:432
3117999323,16cyclictest87350irq/126-enp1s0-08:20:010
3117999322,21cyclictest0-21swapper/012:30:210
3117999321,28cyclictest6920-21/usr/sbin/munin08:40:250
3117999321,27cyclictest808-21snmpd11:08:050
3117999321,21cyclictest0-21swapper/011:40:230
3117999321,19cyclictest2522-21/usr/sbin/munin08:30:260
3117999321,19cyclictest0-21swapper/011:51:260
3117999321,17cyclictest31447-21sed08:20:190
3119699311,11cyclictest322-21ntp_states10:50:233
3119699311,11cyclictest322-21ntp_states10:50:233
3119699311,10cyclictest0-21swapper/310:39:273
3119299310,10cyclictest0-21swapper/212:11:232
3117999312,25cyclictest20562-21ntp_states07:55:220
3117999311,26cyclictest10183-40dnf07:31:230
3117999311,20cyclictest6350-21awk12:20:010
3117999311,18cyclictest808-21snmpd08:57:390
3117999311,18cyclictest21686-21/usr/sbin/munin08:00:160
3117999311,17cyclictest24900-21munin-run10:35:020
3117999310,4cyclictest0-21swapper/010:00:070
3119699308,3cyclictest0-21swapper/310:15:193
3119699301,3cyclictest0-21swapper/311:56:193
3119699301,13cyclictest0-21swapper/308:57:473
3119699301,13cyclictest0-21swapper/308:04:153
3119699301,12cyclictest0-21swapper/312:33:313
3119699301,12cyclictest0-21swapper/307:28:113
3119699301,11cyclictest0-21swapper/311:06:553
3119299301,9cyclictest0-21swapper/210:33:232
3119299301,26cyclictest0-21swapper/209:23:392
3119299301,19cyclictest0-21swapper/208:18:192
3119299301,11cyclictest0-21swapper/210:42:152
3119299301,11cyclictest0-21swapper/210:13:472
3119299300,11cyclictest0-21swapper/211:15:152
3118599301,8cyclictest0-21swapper/108:02:051
3117999304,19cyclictest87350irq/126-enp1s0-09:45:120
3117999302,25cyclictest18951-21cut10:20:170
3117999302,24cyclictest4724-21/usr/sbin/munin08:35:120
3117999301,4cyclictest3660-21cat11:00:010
3117999300,1cyclictest0-21swapper/007:45:260
3119699291,9cyclictest0-21swapper/307:15:553
3119699291,17cyclictest0-21swapper/310:49:153
3119699291,13cyclictest0-21swapper/308:41:273
3119699291,12cyclictest0-21swapper/311:18:193
3119699291,12cyclictest0-21swapper/310:42:073
3119699291,11cyclictest0-21swapper/309:25:393
3119699291,10cyclictest0-21swapper/307:39:193
3119299291,9cyclictest0-21swapper/210:59:192
3119299291,9cyclictest0-21swapper/210:53:392
3119299291,9cyclictest0-21swapper/210:53:392
3119299291,9cyclictest0-21swapper/210:37:152
3119299291,9cyclictest0-21swapper/210:06:072
3119299291,8cyclictest0-21swapper/212:37:552
3119299291,8cyclictest0-21swapper/212:07:472
3119299291,10cyclictest0-21swapper/211:31:032
3119299291,10cyclictest0-21swapper/210:49:312
3119299291,10cyclictest0-21swapper/208:36:072
3117999299,17cyclictest28145-21date08:15:010
3117999296,20cyclictest1-21systemd10:10:200
3117999292,13cyclictest3902-21perf07:20:010
3117999291,4cyclictest16460-21sshd11:26:540
3117999291,20cyclictest766-21ntpd08:46:150
3119699288,3cyclictest24033-21taskset09:16:593
3119699281,9cyclictest0-21swapper/311:22:073
3119699281,8cyclictest0-21swapper/312:03:473
3119699281,8cyclictest0-21swapper/312:03:473
3119699281,18cyclictest0-21swapper/308:14:153
3119699281,12cyclictest0-21swapper/311:46:433
3119699281,12cyclictest0-21swapper/309:00:313
3119699281,12cyclictest0-21swapper/308:47:313
3119699281,12cyclictest0-21swapper/308:16:153
3119699281,12cyclictest0-21swapper/307:40:393
3119699281,11cyclictest0-21swapper/312:27:313
3119699281,10cyclictest0-21swapper/312:15:323
3119699281,10cyclictest0-21swapper/308:53:073
3119699280,26cyclictest0-21swapper/309:38:593
3119699280,26cyclictest0-21swapper/309:38:593
3119699280,19cyclictest0-21swapper/309:47:073
3119699280,11cyclictest0-21swapper/309:54:353
3119699280,10cyclictest0-21swapper/308:27:213
3119699280,10cyclictest0-21swapper/308:27:213
3119299287,3cyclictest0-21swapper/207:11:512
3119299287,3cyclictest0-21swapper/207:11:512
3119299284,3cyclictest0-21swapper/208:14:092
3119299281,9cyclictest0-21swapper/208:45:192
3119299281,8cyclictest0-21swapper/207:45:112
3119299281,8cyclictest0-21swapper/207:15:152
3119299281,12cyclictest0-21swapper/208:09:592
3119299281,10cyclictest0-21swapper/209:01:512
3119299280,12cyclictest0-21swapper/209:11:192
3117999288,17cyclictest15894-21cstates09:00:140
3117999282,23cyclictest10705-21/usr/sbin/munin07:35:160
3117999281,4cyclictest17294-21/usr/sbin/munin07:50:280
3117999281,4cyclictest1447-21/usr/sbin/munin09:40:150
3117999281,24cyclictest18391-21uniq09:05:140
3117999281,23cyclictest12388-21ntp_states08:50:230
3117999280,2cyclictest0-21swapper/009:55:200
3119699278,3cyclictest11321-21users10:00:293
3119699271,12cyclictest0-21swapper/311:40:353
3119699271,11cyclictest0-21swapper/311:00:513
3119699271,11cyclictest0-21swapper/310:57:393
3119699271,11cyclictest0-21swapper/307:22:113
3119699271,10cyclictest0-21swapper/307:50:433
3119299271,8cyclictest0-21swapper/211:28:152
3119299271,7cyclictest0-21swapper/209:05:182
3119299271,17cyclictest0-21swapper/208:00:132
3119299271,12cyclictest0-21swapper/208:27:272
3119299271,12cyclictest0-21swapper/208:27:272
3119299271,10cyclictest0-21swapper/211:47:562
3119299271,10cyclictest0-21swapper/210:21:552
3119299271,10cyclictest0-21swapper/208:34:432
3119299270,11cyclictest0-21swapper/212:15:192
3117999270,2cyclictest32684-21/usr/sbin/munin08:25:260
3117999270,2cyclictest32684-21/usr/sbin/munin08:25:260
3119699261,9cyclictest0-21swapper/309:20:313
3119699261,11cyclictest0-21swapper/310:13:233
3119699261,10cyclictest0-21swapper/309:41:483
3119699260,13cyclictest0-21swapper/309:09:533
3119299261,10cyclictest0-21swapper/208:23:072
3119299260,8cyclictest0-21swapper/207:38:032
3118599264,10cyclictest0-21swapper/110:45:371
3118599261,17cyclictest29193-21/usr/sbin/munin11:55:241
3119699259,2cyclictest0-21swapper/310:25:113
3119699251,11cyclictest0-21swapper/308:07:473
3119699251,10cyclictest0-21swapper/311:53:033
3119299257,3cyclictest0-21swapper/211:56:332
3119299251,9cyclictest0-21swapper/212:32:052
3119299251,9cyclictest0-21swapper/208:42:372
3119299251,8cyclictest0-21swapper/211:37:592
3119299251,8cyclictest0-21swapper/207:25:192
3119299251,7cyclictest0-21swapper/207:22:392
3119699241,7cyclictest0-21swapper/309:30:513
3119699241,11cyclictest0-21swapper/310:33:553
3119299241,8cyclictest0-21swapper/207:31:032
3119299240,9cyclictest0-21swapper/211:51:072
3118599241,18cyclictest0-21swapper/112:08:181
3118599240,19cyclictest0-21swapper/110:06:381
3119699235,3cyclictest0-21swapper/308:21:433
3119699231,3cyclictest0-21swapper/307:55:033
3119699230,9cyclictest0-21swapper/312:11:243
3119299230,10cyclictest0-21swapper/208:55:172
3118599232,13cyclictest0-21swapper/109:25:491
3118599231,8cyclictest0-21swapper/112:26:551
3118599231,7cyclictest0-21swapper/109:59:001
3118599231,3cyclictest0-21swapper/110:35:011
3118599231,19cyclictest10690-21df11:15:141
3118599227,8cyclictest0-21swapper/108:50:241
3118599222,11cyclictest7254-21df08:40:121
3118599221,9cyclictest0-21swapper/112:13:101
31185992213,2cyclictest31208-21uname12:00:221
31185992213,2cyclictest31208-21uname12:00:221
31185992213,2cyclictest11604-21sshd12:30:141
3118599221,18cyclictest0-21swapper/111:35:201
3118599221,16cyclictest0-21swapper/111:04:511
3118599221,15cyclictest0-21swapper/108:49:201
3118599221,15cyclictest0-21swapper/107:23:161
31185992211,3cyclictest15662-21memory11:25:181
3118599221,11cyclictest18305-21munin-run10:20:011
3118599220,16cyclictest0-21swapper/109:05:111
3119699216,3cyclictest0-21swapper/308:30:393
3119699212,3cyclictest87650irq/129-enp1s0-11:27:023
3119699212,2cyclictest89050irq/133-enp2s0-07:13:093
3119699212,2cyclictest89050irq/133-enp2s0-07:13:093
3119699210,8cyclictest0-21swapper/309:10:473
3119299214,3cyclictest0-21swapper/209:41:132
3118599219,3cyclictest25886-21grep11:50:131
3118599216,9cyclictest0-21swapper/111:10:591
3118599213,9cyclictest21-21rcuc/110:10:151
3118599211,8cyclictest0-21swapper/109:22:371
3118599211,5cyclictest0-21swapper/109:11:501
3118599211,3cyclictest0-21swapper/109:45:401
3118599211,15cyclictest0-21swapper/111:46:391
3118599211,15cyclictest0-21swapper/111:43:431
3118599211,15cyclictest0-21swapper/111:22:111
3118599211,15cyclictest0-21swapper/111:08:141
3118599211,15cyclictest0-21swapper/109:19:591
3118599211,15cyclictest0-21swapper/108:13:141
3118599211,15cyclictest0-21swapper/107:57:261
3118599211,15cyclictest0-21swapper/107:53:261
3118599211,11cyclictest8283-21munin-run07:30:011
3118599211,11cyclictest32451-21latency_hist08:25:011
3118599211,11cyclictest15908-21irqstats07:45:171
3118599211,11cyclictest13817-21cat08:55:141
3118599210,15cyclictest0-21swapper/109:00:081
3118599210,10cyclictest0-21swapper/110:25:101
102562211,3sleep1808-21snmpd07:32:191
3118599209,3cyclictest4498-21munin-run08:35:011
3118599208,4cyclictest32691-21users09:35:281
3118599208,4cyclictest32691-21users09:35:281
3118599208,4cyclictest221ktimersoftd/110:00:281
3118599207,5cyclictest5653-21switchtime12:15:241
3118599204,9cyclictest0-21swapper/108:25:181
3118599204,9cyclictest0-21swapper/108:25:181
3118599201,9cyclictest0-21swapper/109:52:271
3118599201,3cyclictest0-21swapper/108:38:031
31185992012,3cyclictest0-21swapper/112:22:031
3118599201,17cyclictest0-21swapper/110:20:221
3118599201,14cyclictest0-21swapper/110:36:111
3118599201,14cyclictest0-21swapper/108:08:001
31185992011,3cyclictest0-21swapper/110:55:361
3118599201,12cyclictest0-21swapper/107:40:131
3118599201,11cyclictest976-21awk07:15:011
3118599201,11cyclictest976-21awk07:15:011
3118599201,10cyclictest0-21swapper/112:35:131
3118599194,5cyclictest87650irq/129-enp1s0-09:40:051
3118599191,14cyclictest0-21swapper/107:39:121
3118599191,10cyclictest28760-21sed10:40:251
31185991910,5cyclictest0-21swapper/108:17:131
3118599190,15cyclictest0-21swapper/109:33:261
3118599190,14cyclictest0-21swapper/110:50:291
3118599190,14cyclictest0-21swapper/110:50:291
31185991812,1cyclictest0-21swapper/107:18:131
178602181,2sleep10-21swapper/111:30:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional