You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 23:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Sep 16, 2025 12:44:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
194458621180,4sleep20-21swapper/209:35:122
162399890,68rtkit-daemon0-21swapper/207:09:512
19254462870,8sleep32197014-21PK-Backend08:40:033
19039742810,6sleep01753-21gnome-shell07:40:020
19039742810,6sleep01753-21gnome-shell07:40:020
189254327555,13sleep00-21swapper/007:06:540
19014832710,8sleep32197014-21PK-Backend07:30:363
189252127145,17sleep30-21swapper/307:06:373
19588632660,14sleep1251rcuc/110:15:111
19316612650,6sleep21931660-21sshd08:55:312
19316612650,6sleep21931660-21sshd08:55:312
189208526448,9sleep10-21swapper/107:05:231
1892940996322,37cyclictest1622-21rtkit-daemon10:05:510
162399580,8rtkit-daemon1622-21rtkit-daemon09:21:510
19027532570,9sleep31753-21gnome-shell07:35:183
19027532570,9sleep31753-21gnome-shell07:35:183
162399570,9rtkit-daemon1622-21rtkit-daemon11:46:560
162399570,8rtkit-daemon1622-21rtkit-daemon12:33:270
162399570,8rtkit-daemon1622-21rtkit-daemon12:10:480
162399570,8rtkit-daemon1622-21rtkit-daemon09:07:210
1892950995610,18cyclictest42-21ksoftirqd/309:35:223
1892940995623,10cyclictest13-21ksoftirqd/012:25:010
19895592550,7sleep10-21swapper/111:40:281
162399550,8rtkit-daemon1622-21rtkit-daemon09:29:390
19655262540,6sleep22197014-21PK-Backend10:33:032
19655262540,6sleep22197014-21PK-Backend10:33:032
189295099547,10cyclictest42-21ksoftirqd/310:05:193
1892940995422,10cyclictest13-21ksoftirqd/011:05:010
162399530,8rtkit-daemon1622-21rtkit-daemon09:32:281
1892950995216,19cyclictest42-21ksoftirqd/310:33:033
1892950995216,19cyclictest42-21ksoftirqd/310:33:033
1892940995225,8cyclictest13-21ksoftirqd/009:30:280
189294099521,26cyclictest1926390-21chrt08:40:270
1892940995212,21cyclictest13-21ksoftirqd/007:25:010
162399520,8rtkit-daemon1622-21rtkit-daemon09:35:290
1892950995111,18cyclictest42-21ksoftirqd/308:16:483
1892940995121,10cyclictest13-21ksoftirqd/012:27:440
1892940995112,18cyclictest13-21ksoftirqd/011:30:190
1892940995112,18cyclictest13-21ksoftirqd/011:30:190
189295099502,9cyclictest1753-21gnome-shell09:50:333
1892950995014,8cyclictest42-21ksoftirqd/307:21:163
1892940995022,24cyclictest1996634-40lsb_release12:00:270
1892940995013,18cyclictest13-21ksoftirqd/010:35:020
1892940995013,18cyclictest13-21ksoftirqd/010:35:020
1892950994927,18cyclictest1959678-21/usr/sbin/munin10:15:283
1892948994923,21cyclictest1895973-21missed_timers07:15:262
189294099499,20cyclictest13-21ksoftirqd/010:20:130
189294099499,18cyclictest13-21ksoftirqd/007:55:150
1892940994917,11cyclictest13-21ksoftirqd/008:18:080
1892940994912,19cyclictest13-21ksoftirqd/011:25:010
1892940994911,19cyclictest13-21ksoftirqd/008:57:520
1892940994911,19cyclictest13-21ksoftirqd/008:57:510
18959742480,9chrt2197014-21PK-Backend07:15:260
1892950994815,9cyclictest42-21ksoftirqd/308:05:123
1892940994824,20cyclictest1947096-21ntp_states09:40:300
1892940994818,9cyclictest13-21ksoftirqd/012:09:360
189294099481,12cyclictest13-21ksoftirqd/011:30:010
19420092470,5sleep2331rcuc/209:25:292
1892950994710,17cyclictest42-21ksoftirqd/308:48:483
1892948994733,10cyclictest1955190-21expr10:05:192
1892940994725,18cyclictest1966039-21grep10:35:150
1892940994721,6cyclictest141rcu_preempt11:16:140
1892940994718,9cyclictest13-21ksoftirqd/011:14:010
1892940994712,18cyclictest13-21ksoftirqd/011:10:010
189294099471,12cyclictest13-21ksoftirqd/008:45:140
162399470,33rtkit-daemon2197014-21PK-Backend07:11:363
19196492460,8sleep02197014-21PK-Backend08:20:310
19196492460,8sleep02197014-21PK-Backend08:20:310
1892950994625,5cyclictest141rcu_preempt09:15:193
1892950994615,9cyclictest1753-21gnome-shell09:03:093
1892950994615,9cyclictest1753-21gnome-shell09:03:093
1892943994612,5cyclictest141rcu_preempt07:21:191
189294099467,18cyclictest13-21ksoftirqd/009:10:150
189294099466,4cyclictest141rcu_preempt10:30:010
189294099466,20cyclictest13-21ksoftirqd/007:40:170
189294099466,20cyclictest13-21ksoftirqd/007:40:160
1892940994633,9cyclictest13-21ksoftirqd/010:00:300
1892940994622,6cyclictest141rcu_preempt07:13:030
1892940994610,4cyclictest141rcu_preempt09:20:010
162399460,35rtkit-daemon1069-21runrttasks11:37:233
19837552450,5sleep10-21swapper/111:25:151
189295099455,24cyclictest141rcu_preempt10:21:443
189295099453,19cyclictest1753-21gnome-shell08:40:213
189294899459,11cyclictest141rcu_preempt09:42:362
1892948994512,11cyclictest141rcu_preempt07:55:012
1892940994524,17cyclictest2001885-21ntpq12:15:280
189294099452,12cyclictest13-21ksoftirqd/007:25:280
1892940994520,6cyclictest141rcu_preempt12:39:310
1892940994520,6cyclictest141rcu_preempt12:39:310
1892940994516,9cyclictest13-21ksoftirqd/008:13:510
1892940994514,9cyclictest13-21ksoftirqd/008:07:180
1892950994413,5cyclictest141rcu_preempt11:10:153
1892950994411,9cyclictest42-21ksoftirqd/311:01:313
1892950994410,16cyclictest42-21ksoftirqd/312:04:553
1892950994410,16cyclictest42-21ksoftirqd/308:51:483
1892948994421,19cyclictest1979978-40apt-check11:15:132
1892948994415,7cyclictest141rcu_preempt11:43:022
1892943994413,6cyclictest141rcu_preempt08:10:151
189294099448,16cyclictest1957769-21strings10:10:240
189294099446,18cyclictest13-21ksoftirqd/008:50:140
1892940994415,9cyclictest13-21ksoftirqd/011:35:270
1892940994412,9cyclictest13-21ksoftirqd/007:45:310
162399440,5rtkit-daemon26-21ksoftirqd/112:15:131
162399440,31rtkit-daemon2197014-21PK-Backend08:43:092
189295099438,15cyclictest1753-21gnome-shell08:20:133
189295099438,15cyclictest1753-21gnome-shell08:20:133
189295099436,13cyclictest141rcu_preempt11:22:513
1892950994333,6cyclictest411rcuc/309:20:033
189295099433,21cyclictest42-21ksoftirqd/309:45:023
1892950994326,12cyclictest2000846-21cron12:15:003
1892950994319,5cyclictest141rcu_preempt11:52:463
1892950994319,5cyclictest141rcu_preempt11:52:463
1892950994318,5cyclictest42-21ksoftirqd/308:27:183
1892950994315,9cyclictest42-21ksoftirqd/310:29:493
1892950994310,9cyclictest42-21ksoftirqd/311:57:523
1892948994325,13cyclictest1923213-21sshd08:31:272
1892943994327,12cyclictest1901881-21sshd07:33:281
189294099436,15cyclictest1993891-21apt11:55:120
189294099434,16cyclictest13-21ksoftirqd/007:50:270
189294099433,21cyclictest13-21ksoftirqd/010:51:300
1892940994331,9cyclictest13-21ksoftirqd/008:32:440
189294099432,22cyclictest13-21ksoftirqd/011:55:010
189294099432,22cyclictest13-21ksoftirqd/011:55:010
189294099432,20cyclictest13-21ksoftirqd/007:34:530
1892940994320,9cyclictest13-21ksoftirqd/010:44:500
1892940994319,5cyclictest141rcu_preempt08:00:230
1892940994313,9cyclictest13-21ksoftirqd/010:15:280
189294099431,18cyclictest13-21ksoftirqd/011:45:020
189294099431,17cyclictest13-21ksoftirqd/009:50:230
189294099431,13cyclictest13-21ksoftirqd/008:25:140
189294099431,12cyclictest13-21ksoftirqd/009:50:020
162399430,8rtkit-daemon1753-21gnome-shell09:08:271
19065262420,17sleep126-21ksoftirqd/107:45:221
189295099428,5cyclictest141rcu_preempt07:50:293
189295099428,14cyclictest141rcu_preempt10:40:023
189295099427,19cyclictest42-21ksoftirqd/307:40:283
189295099427,19cyclictest42-21ksoftirqd/307:40:273
189295099426,19cyclictest42-21ksoftirqd/309:50:003
189295099421,15cyclictest2197014-21PK-Backend09:59:253
189294899429,11cyclictest141rcu_preempt07:45:172
1892948994215,6cyclictest141rcu_preempt10:53:242
1892948994215,11cyclictest141rcu_preempt07:35:292
1892948994215,11cyclictest141rcu_preempt07:35:292
1892948994214,7cyclictest141rcu_preempt10:58:102
189294099425,19cyclictest13-21ksoftirqd/010:55:270
162399420,35rtkit-daemon2197014-21PK-Backend11:49:531
162399420,30rtkit-daemon1622-21rtkit-daemon09:40:301
162399420,29rtkit-daemon0-21swapper/310:42:243
20016192410,7sleep22197014-21PK-Backend12:15:232
189295099418,16cyclictest42-21ksoftirqd/311:07:393
189295099415,5cyclictest141rcu_preempt09:05:353
189295099415,4cyclictest141rcu_preempt09:30:003
189295099413,20cyclictest42-21ksoftirqd/312:30:503
1892950994129,8cyclictest42-21ksoftirqd/308:05:013
189295099412,34cyclictest1753-21gnome-shell12:25:093
1892950994122,16cyclictest42-21ksoftirqd/307:46:503
1892950994121,13cyclictest42-21ksoftirqd/312:35:203
1892950994121,13cyclictest42-21ksoftirqd/312:35:203
1892950994119,5cyclictest141rcu_preempt10:52:503
1892950994117,12cyclictest42-21ksoftirqd/307:15:203
1892950994116,22cyclictest42-21ksoftirqd/312:05:173
189294899419,13cyclictest141rcu_preempt08:35:282
1892948994122,15cyclictest1919303-21ntpq08:20:252
1892948994122,15cyclictest1919303-21ntpq08:20:242
1892948994115,6cyclictest141rcu_preempt11:59:242
1892948994115,5cyclictest141rcu_preempt09:10:562
1892948994114,6cyclictest141rcu_preempt12:11:212
1892948994114,6cyclictest141rcu_preempt12:00:202
1892948994114,6cyclictest141rcu_preempt10:00:522
1892948994114,6cyclictest141rcu_preempt08:30:012
1892948994112,6cyclictest141rcu_preempt12:26:512
189294399418,6cyclictest141rcu_preempt09:00:311
189294399418,6cyclictest141rcu_preempt09:00:301
1892943994126,10cyclictest1893620-21cut07:10:221
1892943994111,26cyclictest1914848-21unin-run08:10:011
1892943994110,6cyclictest141rcu_preempt12:39:551
1892943994110,6cyclictest141rcu_preempt12:39:551
1892940994110,8cyclictest13-21ksoftirqd/008:36:350
162399410,5rtkit-daemon0-21swapper/311:18:423
162399410,29rtkit-daemon0-21swapper/211:37:062
189295099409,5cyclictest141rcu_preempt10:10:243
1892950994017,8cyclictest1753-21gnome-shell10:58:173
1892950994016,9cyclictest42-21ksoftirqd/311:43:253
1892950994016,15cyclictest42-21ksoftirqd/311:32:003
1892950994016,15cyclictest42-21ksoftirqd/311:32:003
1892950994015,14cyclictest42-21ksoftirqd/312:23:013
1892950994013,5cyclictest141rcu_preempt11:47:153
189295099401,15cyclictest42-21ksoftirqd/310:00:303
1892948994014,6cyclictest141rcu_preempt12:38:502
1892948994014,6cyclictest141rcu_preempt12:38:492
1892948994014,6cyclictest141rcu_preempt12:23:062
1892948994014,6cyclictest141rcu_preempt11:35:032
1892948994014,6cyclictest141rcu_preempt11:35:022
1892948994014,6cyclictest141rcu_preempt10:48:312
1892948994014,6cyclictest141rcu_preempt08:50:522
1892948994014,6cyclictest141rcu_preempt08:05:042
1892948994014,5cyclictest141rcu_preempt09:20:112
1892948994013,6cyclictest141rcu_preempt11:23:302
1892948994013,6cyclictest141rcu_preempt07:24:082
1892948994012,7cyclictest141rcu_preempt09:02:362
1892948994012,7cyclictest141rcu_preempt09:02:352
1892943994033,4cyclictest64850irq/126-i91507:25:051
189294399402,34cyclictest1999765-21df_inode12:10:171
1892943994022,13cyclictest1924484-21ntp_kernel_err08:35:241
1892943994020,16cyclictest1951918-21/usr/sbin/munin09:55:171
1892943994018,10cyclictest1980286-21expr11:15:161
1892943994012,11cyclictest141rcu_preempt10:30:141
1892943994012,11cyclictest141rcu_preempt10:30:141
1892940994014,12cyclictest13-21ksoftirqd/009:59:000
189294099401,20cyclictest1932730-21cstates09:00:150
189294099401,20cyclictest1932730-21cstates09:00:150
1892940994011,9cyclictest13-21ksoftirqd/010:46:560
162399400,32rtkit-daemon26-21ksoftirqd/109:27:461
162399400,32rtkit-daemon1922415-21diskstats08:30:181
19480672390,6chrt2197014-21PK-Backend09:45:131
189295099395,16cyclictest0-21swapper/308:30:333
1892950993916,5cyclictest141rcu_preempt07:28:513
1892950993916,4cyclictest42-21ksoftirqd/308:58:063
1892950993916,4cyclictest42-21ksoftirqd/308:58:063
1892950993914,8cyclictest42-21ksoftirqd/312:16:523
1892950993913,10cyclictest141rcu_preempt11:25:123
189294899392,8cyclictest1753-21gnome-shell10:10:202
189294899392,32cyclictest1753-21gnome-shell11:54:482
189294899392,32cyclictest1753-21gnome-shell11:54:472
1892948993922,12cyclictest1976534-21ls11:05:162
1892948993921,13cyclictest855-21dbus-daemon07:42:272
1892948993921,13cyclictest855-21dbus-daemon07:42:272
1892948993914,6cyclictest141rcu_preempt10:38:102
1892948993914,6cyclictest141rcu_preempt09:53:292
1892948993914,6cyclictest141rcu_preempt08:01:372
1892948993913,6cyclictest141rcu_preempt10:25:232
1892948993912,10cyclictest141rcu_preempt07:55:202
1892948993911,6cyclictest141rcu_preempt10:24:502
1892948993911,10cyclictest141rcu_preempt11:15:002
189294399398,15cyclictest141rcu_preempt11:05:131
189294399395,30cyclictest1938518-21cut09:15:281
189294399392,8cyclictest1753-21gnome-shell07:15:241
189294399392,34cyclictest0-21swapper/109:14:131
1892943993922,13cyclictest855-21dbus-daemon12:31:561
1892943993914,6cyclictest141rcu_preempt08:20:301
1892943993914,6cyclictest141rcu_preempt08:20:291
1892943993912,6cyclictest141rcu_preempt10:45:321
1892943993912,12cyclictest141rcu_preempt11:55:031
1892943993912,12cyclictest141rcu_preempt11:55:021
189294399391,20cyclictest904-21systemd-logind10:05:201
162399390,26rtkit-daemon0-21swapper/307:55:073
19318912382,32sleep11753-21gnome-shell08:56:351
19318912382,32sleep11753-21gnome-shell08:56:351
1892950993829,5cyclictest0-21swapper/309:30:593
1892950993813,9cyclictest42-21ksoftirqd/308:13:543
189294899389,10cyclictest141rcu_preempt09:05:242
189294899382,32cyclictest1753-21gnome-shell08:14:002
189294899382,20cyclictest1753-21gnome-shell07:25:132
1892948993818,14cyclictest1974980-21interrupts11:00:222
1892948993814,6cyclictest141rcu_preempt09:30:092
1892948993814,5cyclictest141rcu_preempt12:05:122
1892948993812,6cyclictest141rcu_preempt10:15:302
1892948993812,6cyclictest141rcu_preempt09:55:552
1892948993812,10cyclictest141rcu_preempt07:35:012
1892948993811,10cyclictest141rcu_preempt10:40:342
1892948993810,11cyclictest141rcu_preempt09:15:282
1892948993810,11cyclictest141rcu_preempt08:45:252
1892943993822,12cyclictest1911763-21cat08:00:201
1892943993819,16cyclictest2004570-21/usr/sbin/munin12:25:191
1892943993817,17cyclictest1975142-21cat11:00:241
1892943993816,17cyclictest1979053-21uptime11:10:331
189294399381,33cyclictest1753-21gnome-shell12:00:011
1892943993811,6cyclictest141rcu_preempt12:23:481
162399380,26rtkit-daemon1622-21rtkit-daemon08:29:241
189295099376,18cyclictest42-21ksoftirqd/310:45:253
189295099376,14cyclictest141rcu_preempt09:15:023
1892948993726,7cyclictest1753-21gnome-shell11:30:012
1892948993714,6cyclictest141rcu_preempt09:46:552
1892948993714,6cyclictest141rcu_preempt08:15:332
1892948993712,5cyclictest141rcu_preempt07:10:572
1892948993711,6cyclictest141rcu_preempt12:32:332
1892948993710,11cyclictest141rcu_preempt11:45:312
189294399378,12cyclictest141rcu_preempt07:35:261
189294399378,12cyclictest141rcu_preempt07:35:251
1892943993728,5cyclictest356-21jbd2/sdb2-810:25:181
1892943993727,7cyclictest1069-21runrttasks07:57:161
1892943993720,12cyclictest884-21gdbus09:55:011
1892943993718,14cyclictest3242253-21nautilus10:55:021
1892943993715,18cyclictest1069-21runrttasks10:24:001
1892943993712,6cyclictest141rcu_preempt09:35:311
1892943993712,6cyclictest141rcu_preempt09:20:301
1892943993711,6cyclictest141rcu_preempt12:00:121
1892943993711,5cyclictest141rcu_preempt08:55:021
189294399367,5cyclictest141rcu_preempt11:40:021
1892943993622,10cyclictest26-21ksoftirqd/111:30:341
1892943993622,10cyclictest26-21ksoftirqd/111:30:331
1892943993619,13cyclictest1954022-21ntpq10:00:261
1892943993619,12cyclictest1957864-21awk10:10:291
1892943993612,5cyclictest141rcu_preempt10:55:231
1892943993612,21cyclictest1997830-21cpuspeed_turbos12:05:161
1892943993520,11cyclictest26-21ksoftirqd/108:42:321
1892943993520,11cyclictest1928815-21date08:50:021
1892943993519,11cyclictest1966695-21ntp_kernel_err10:35:251
1892943993511,6cyclictest141rcu_preempt07:40:301
1892943993511,6cyclictest141rcu_preempt07:40:291
189294399342,11cyclictest1917573-21ntp_states08:15:261
1892943993420,10cyclictest26-21ksoftirqd/110:40:181
1892943993418,11cyclictest64850irq/126-i91511:20:021
1892943993317,12cyclictest1908567-21awk07:50:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional