You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-07 - 14:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Sat Mar 07, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4140875213560,70sleep30-21swapper/307:09:543
414058428063,11sleep10-21swapper/107:06:151
414053728061,12sleep20-21swapper/207:05:402
414066827451,17sleep00-21swapper/007:07:180
1927602241,7sleep0192973-21diskstats10:15:120
3418862230,3sleep2341866-21munin-run11:40:002
414101199221,17cyclictest289000-21taskset11:09:583
414101199212,14cyclictest3575992sleep311:47:203
414101199211,17cyclictest419903-21taskset12:23:063
414101199210,19cyclictest0-21swapper/312:00:193
4141011992011,6cyclictest190635-21taskset10:12:503
414101199201,15cyclictest0-21swapper/312:36:353
414101199201,14cyclictest0-21swapper/312:05:243
1731292201,7sleep277587-21diskmemload10:02:562
4141011991915,2cyclictest0-21swapper/311:20:173
4141011991915,2cyclictest0-21swapper/310:47:463
414101199191,15cyclictest233114-21taskset10:35:473
414101199191,14cyclictest226481-21taskset10:33:383
414101199191,14cyclictest109575-21taskset09:25:323
414101199191,14cyclictest0-21swapper/312:10:193
414101199191,14cyclictest0-21swapper/311:40:043
414101199191,14cyclictest0-21swapper/310:56:413
414101199191,14cyclictest0-21swapper/310:43:173
414101199190,16cyclictest0-21swapper/309:36:013
414101199190,15cyclictest259610-21taskset10:50:583
1208202191,3sleep277587-21diskmemload09:33:492
414101199181,7cyclictest435670-21taskset12:30:483
414101199181,4cyclictest340080-21taskset11:37:453
414101199181,4cyclictest0-21swapper/310:22:273
414101199181,2cyclictest0-21swapper/309:48:413
414101199181,14cyclictest92754-21chrt09:16:303
414101199181,14cyclictest0-21swapper/312:26:003
414101199181,14cyclictest0-21swapper/312:17:363
414101199181,14cyclictest0-21swapper/311:32:573
414101199181,14cyclictest0-21swapper/311:00:323
414101199181,14cyclictest0-21swapper/310:26:063
414101199181,14cyclictest0-21swapper/310:06:273
414101199181,14cyclictest0-21swapper/310:01:163
4141011991811,3cyclictest0-21swapper/311:10:343
414101199181,13cyclictest0-21swapper/309:50:063
414101199181,13cyclictest0-21swapper/308:50:183
414101199180,15cyclictest0-21swapper/309:22:573
3931342181,3sleep0393145-21ssh12:07:290
4141011991711,4cyclictest66950irq/132-enp1s0-TxRx-309:30:213
414101199171,14cyclictest0-21swapper/311:55:013
414101199171,13cyclictest0-21swapper/311:25:143
414101199171,13cyclictest0-21swapper/311:15:203
414101199171,13cyclictest0-21swapper/310:15:113
414101199171,13cyclictest0-21swapper/309:56:053
414101199171,13cyclictest0-21swapper/309:56:053
414101199171,13cyclictest0-21swapper/309:40:083
4141011991711,3cyclictest0-21swapper/309:12:143
414101199170,3cyclictest23687-21taskset08:19:463
414101199170,13cyclictest0-21swapper/311:50:103
3466462172,5sleep077587-21diskmemload11:40:190
1197132171,3sleep10-21swapper/109:32:201
414101199169,4cyclictest12923-21chrt08:08:243
414101199161,12cyclictest0-21swapper/307:15:123
414101199160,13cyclictest0-21swapper/308:35:143
414101199160,13cyclictest0-21swapper/308:30:183
414101199160,13cyclictest0-21swapper/308:20:213
414101199160,13cyclictest0-21swapper/308:10:253
414101199160,12cyclictest0-21swapper/307:40:193
4111452161,4sleep0411159-21ssh12:18:110
3935752161,5sleep277587-21diskmemload12:08:032
3778682160,1sleep10-21swapper/112:00:081
3059302161,3sleep177587-21diskmemload11:19:071
2897082162,7sleep1289843-21perl11:10:131
2507962161,3sleep00-21swapper/010:45:550
2409612161,10sleep2241021-21ntpq10:40:202
2267612161,4sleep077587-21diskmemload10:34:010
948732151,2sleep20-21swapper/209:19:182
414101199151,3cyclictest0-21swapper/308:40:113
414101199151,13cyclictest0-21swapper/307:30:163
4141011991511,1cyclictest0-21swapper/309:10:003
414101199151,11cyclictest0-21swapper/307:20:133
414101199150,1cyclictest0-21swapper/307:15:003
414101199150,13cyclictest0-21swapper/307:35:263
414101199150,12cyclictest0-21swapper/309:00:023
414101199150,12cyclictest0-21swapper/308:55:253
414101199150,12cyclictest0-21swapper/307:55:213
414101199150,12cyclictest0-21swapper/307:45:223
3750382151,4sleep0375053-21ssh11:56:470
3555142151,8sleep1355579-21ntpq11:45:211
2959262152,3sleep277587-21diskmemload11:12:312
2877052151,8sleep177587-21diskmemload11:08:171
2768342151,6sleep00-21swapper/011:00:300
1905122151,4sleep1190523-21ssh10:12:411
1646242150,1sleep10-21swapper/109:58:181
1646242150,1sleep10-21swapper/109:58:181
1482332151,3sleep177587-21diskmemload09:49:521
1222992150,3sleep177587-21diskmemload09:35:051
1106282151,4sleep277587-21diskmemload09:26:562
1024052150,1sleep20-21swapper/209:22:392
4443472141,2sleep00-21swapper/012:35:430
4141011991411,1cyclictest0-21swapper/308:45:153
414101199140,2cyclictest0-21swapper/307:50:133
414101199140,2cyclictest0-21swapper/307:25:223
414101199140,12cyclictest0-21swapper/308:00:133
414101199140,10cyclictest0-21swapper/308:25:123
3492222141,3sleep277587-21diskmemload11:42:502
3402612141,2sleep10-21swapper/111:37:591
271192141,9sleep228192-21memory08:20:192
2693302141,2sleep177587-21diskmemload10:57:151
2337902140,2sleep00-21swapper/010:36:410
1554172141,2sleep20-21swapper/209:52:432
1373882141,4sleep2137399-21ssh09:42:072
857622131,6sleep277587-21diskmemload09:14:172
856952131,7sleep00-21swapper/009:14:110
681532136,3sleep169853-21grep09:00:181
4305582131,7sleep0430785-21df_inode12:30:140
41591852132,6sleep24159204-21apt-get07:25:122
4034792131,7sleep177587-21diskmemload12:14:391
3819232131,8sleep2382072-21nfs4_client12:00:172
3778532131,8sleep0377867-21ssh12:00:070
2507572130,2sleep177587-21diskmemload10:45:521
2019612131,5sleep1202080-21irqstats10:20:151
1648142131,6sleep00-21swapper/009:58:330
1648142131,6sleep00-21swapper/009:58:330
1476042131,4sleep0147618-21ssh09:49:030
1293632130,1sleep20-21swapper/209:38:062
80542121,6sleep28073-21apt-get08:05:122
4370202121,7sleep177587-21diskmemload12:32:351
41749932120,7sleep04174937-31find07:39:000
41525802121,5sleep04152629-21ntpq07:15:200
3924352121,8sleep1392446-21ssh12:06:341
3475762121,7sleep177587-21diskmemload11:40:401
3224112121,6sleep177587-21diskmemload11:27:381
3202832122,2sleep2171rcu_preempt11:25:262
3050272121,5sleep277587-21diskmemload11:17:562
2784262121,7sleep1278440-21ssh11:02:371
242692121,5sleep00-21swapper/008:20:120
2347082120,6sleep10-21swapper/110:37:541
2106262121,5sleep1210845-21hddtemp_smartct10:25:161
2102962122,4sleep2210321-21apt-get10:25:132
1480352121,6sleep20-21swapper/209:49:372
1102112121,7sleep1110222-21ssh09:26:241
1023232121,3sleep0102337-21ssh09:22:330
775732111,5sleep277627-21munin-run09:10:002
621662111,6sleep263744-21cut08:55:172
562382111,6sleep256256-21apt-get08:50:122
4470782110,5sleep10-21swapper/112:39:171
4370872111,4sleep277587-21diskmemload12:32:402
41913422111,5sleep14191361-21apt-get07:55:131
4187932111,3sleep00-21swapper/012:21:400
41749542111,4sleep14174937-31find07:38:531
41455822111,6sleep24145589-21/usr/sbin/munin07:10:252
4084042111,5sleep2408471-21ntp_states12:15:212
408212111,3sleep041899-21kthreadcore08:35:150
3683052111,5sleep177587-21diskmemload11:54:521
3581792111,3sleep00-21swapper/011:48:050
349222116,2sleep20-21swapper/208:30:082
3377442111,3sleep00-21swapper/011:35:220
3319922111,3sleep20-21swapper/211:33:402
3237732111,5sleep077587-21diskmemload11:29:250
3121842111,4sleep177587-21diskmemload11:20:441
3112422111,5sleep2311289-21ntpq11:20:212
2936672111,5sleep0293726-21ntpq11:10:200
2890152111,5sleep0289073-21munin-run11:10:000
283842111,5sleep128438-21ntpq08:20:221
2714452111,5sleep2271526-21munin-run11:00:002
2619122111,1sleep00-21swapper/010:54:020
2613632111,3sleep10-21swapper/110:53:181
2607062111,4sleep277587-21diskmemload10:52:272
2535742111,4sleep20-21swapper/210:49:342
2429862110,2sleep10-21swapper/110:42:111
2271332111,4sleep177587-21diskmemload10:34:301
2093632111,7sleep0209377-21ssh10:24:180
1751312111,5sleep2175359-21ls10:05:122
1750432111,4sleep177587-21diskmemload10:05:071
1720982111,4sleep177587-21diskmemload10:01:341
1287682111,3sleep0128782-21ssh09:37:200
933872101,3sleep10-21swapper/109:17:211
933332101,3sleep00-21swapper/009:17:160
83302101,6sleep08355-21fschecks_count08:05:140
727872101,4sleep172890-21perl09:05:141
477822101,5sleep049440-21modprobe08:40:160
4448382101,2sleep20-21swapper/212:36:212
4274492101,3sleep00-21swapper/012:26:330
4216392101,3sleep177587-21diskmemload12:25:061
41898522101,5sleep24189970-21missed_timers07:50:172
41804192100,4sleep10-21swapper/107:44:431
41803802103,2sleep2171rcu_preempt07:44:112
41701582101,4sleep24170368-21hddtemp_smartct07:35:142
41541642101,4sleep14154372-21hddtemp_smartct07:20:141
41417612103,3sleep14143247-21grep07:10:161
41411802101,4sleep04141199-21apt-get07:10:120
4132592101,4sleep2413483-21hddtemp_smartct12:20:142
4123852101,4sleep177587-21diskmemload12:19:501
4022042100,1sleep00-21swapper/012:12:580
399932101,4sleep140019-21munin-run08:35:001
3954002100,4sleep20-21swapper/212:10:072
3769062101,4sleep277587-21diskmemload11:59:132
3684212101,5sleep2368439-21apt-get11:55:002
3655892101,6sleep0365601-21ssh11:51:170
3304082101,3sleep177587-21diskmemload11:31:341
3302972100,2sleep00-21swapper/011:31:250
3126712100,2sleep00-21swapper/011:21:230
2866092101,3sleep277587-21diskmemload11:06:502
2791942101,6sleep277587-21diskmemload11:03:382
2673742101,5sleep0267431-21ntpq10:55:290
2190372101,3sleep277587-21diskmemload10:30:062
2179462101,4sleep00-21swapper/010:29:000
2083162100,5sleep277587-21diskmemload10:22:562
189162101,5sleep118966-21cut08:15:131
1888872101,2sleep20-21swapper/210:10:342
1842572101,5sleep0184465-21sed10:10:190
1663812101,4sleep0166651-21df_inode10:00:140
1627922101,3sleep277587-21diskmemload09:55:542
1627922101,3sleep277587-21diskmemload09:55:542
1550462101,3sleep00-21swapper/009:52:150
133132101,4sleep013332-21apt-get08:10:110
1128972101,4sleep077587-21diskmemload09:29:550
86101290,1sleep10-21swapper/109:14:441
73734293,2sleep074600-21kthreadcore09:05:170
56519291,4sleep056635-21grep08:50:140
54971290,4sleep055020-21ntpq08:45:210
45606291,3sleep245625-21apt-get08:40:102
427577291,2sleep20-21swapper/212:26:432
4185777291,4sleep14185793-21ls07:50:001
4185766290,1sleep00-21swapper/007:49:550
4164538291,4sleep14164766-21cstates07:30:121
413134291,3sleep10-21swapper/112:20:131
373451291,4sleep1373496-21ssh11:55:231
354912291,4sleep277587-21diskmemload11:45:182
34648290,1sleep10-21swapper/108:29:511
304135291,4sleep0304150-21ssh11:16:470
244891291,2sleep00-21swapper/010:44:430
233210290,2sleep20-21swapper/210:35:562
197875291,3sleep20-21swapper/210:15:512
197609290,3sleep177587-21diskmemload10:15:291
19630291,4sleep021353-21tr08:15:160
18378291,4sleep118554-21ls08:15:001
180452291,2sleep00-21swapper/010:06:020
148681291,2sleep10-21swapper/109:50:081
137728291,4sleep177587-21diskmemload09:42:341
131200290,5sleep0131211-21ssh09:40:090
115363291,4sleep077587-21diskmemload09:30:190
101270290,3sleep177587-21diskmemload09:21:101
66606281,3sleep067005-21sed09:00:000
45899281,3sleep146113-21smartctl08:40:131
4164516281,2sleep00-21swapper/007:30:110
4153869280,3sleep24153888-21apt-get07:20:122
4146603281,3sleep24148058-21kthreadcore07:15:022
34946281,3sleep035207-21df_inode08:30:100
30546281,3sleep032243-21kthreadcore08:25:170
7447270,1sleep10-21swapper/108:00:381
71992270,1sleep00-21swapper/009:01:490
61051270,1sleep10-21swapper/108:51:521
51189271,3sleep151247-21cut08:45:141
4192298271,2sleep20-21swapper/207:55:182
4190983270,4sleep00-21swapper/007:53:510
41901271,3sleep143680-21kthreadcore08:35:161
4185186270,3sleep20-21swapper/207:45:282
4180168270,1sleep00-21swapper/007:41:290
4154662271,3sleep04156301-21kthreadcore07:20:170
23464270,4sleep20-21swapper/208:16:582
7632260,1sleep20-21swapper/208:02:592
61563260,1sleep10-21swapper/108:55:121
54820260,2sleep254927-21strings08:45:192
44415261,2sleep20-21swapper/208:35:212
4190385261,2sleep10-21swapper/107:50:241
4169365260,1sleep20-21swapper/207:32:012
4163558261,2sleep00-21swapper/007:25:260
4159519260,2sleep10-21swapper/107:25:151
4146277260,2sleep10-21swapper/107:15:011
34665260,2sleep234714-21systemd-run08:30:002
2764261,2sleep00-21swapper/008:00:130
2330260,1sleep00-21swapper/007:58:280
71557250,1sleep20-21swapper/209:00:262
15269250,2sleep20-21swapper/208:10:172
12843250,3sleep10-21swapper/108:07:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional