You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 16:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Nov 19, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
65297922290,10sleep00-21swapper/007:05:260
73312921070,7sleep10-21swapper/110:55:021
6922362930,6sleep2331rcuc/208:55:322
7409422860,5sleep02197014-21PK-Backend11:15:290
65354727839,10sleep20-21swapper/207:08:282
65343527762,9sleep30-21swapper/307:06:593
653823997114,27cyclictest1753-21gnome-shell07:38:453
653823996921,29cyclictest1753-21gnome-shell10:20:173
653823996913,18cyclictest1753-21gnome-shell10:15:013
653823996723,26cyclictest1753-21gnome-shell12:30:163
653823996722,25cyclictest1753-21gnome-shell08:50:143
653823996719,28cyclictest1753-21gnome-shell10:27:153
653823996718,31cyclictest1753-21gnome-shell09:28:543
653823996717,22cyclictest1753-21gnome-shell10:54:493
653823996713,18cyclictest726253-21/usr/sbin/munin10:35:353
653823996712,19cyclictest714850-21grep10:00:253
65260826746,13sleep10-21swapper/107:05:211
6997902660,19sleep0161rcuc/009:20:020
6870692660,5sleep0161rcuc/008:40:330
653823996621,25cyclictest1753-21gnome-shell12:20:253
653823996616,33cyclictest1753-21gnome-shell08:47:143
653823996613,13cyclictest1753-21gnome-shell12:15:273
653823996521,25cyclictest1753-21gnome-shell08:00:283
653823996521,25cyclictest1753-21gnome-shell07:55:213
653823996520,26cyclictest1753-21gnome-shell07:30:023
653823996518,25cyclictest1753-21gnome-shell11:00:353
653823996513,27cyclictest1753-21gnome-shell12:00:483
653823996421,24cyclictest1753-21gnome-shell09:05:153
653823996419,25cyclictest739709-21runrttasks11:12:393
653823996418,27cyclictest1753-21gnome-shell08:29:033
653823996418,21cyclictest1753-21gnome-shell12:39:113
653823996417,28cyclictest1753-21gnome-shell09:59:313
653823996417,26cyclictest1753-21gnome-shell08:10:123
653823996413,29cyclictest1753-21gnome-shell07:50:103
653823996412,15cyclictest1753-21gnome-shell07:40:193
653823996320,25cyclictest1753-21gnome-shell11:45:013
653823996320,22cyclictest1753-21gnome-shell11:20:003
653823996319,27cyclictest1753-21gnome-shell11:47:053
653823996317,27cyclictest1753-21gnome-shell08:20:133
653823996317,26cyclictest1753-21gnome-shell10:30:433
653823996317,26cyclictest1753-21gnome-shell09:16:053
653823996316,28cyclictest759283-40lsb_release12:06:233
653823996313,28cyclictest1753-21gnome-shell11:50:153
653823996313,28cyclictest1753-21gnome-shell11:50:143
653823996312,27cyclictest1753-21gnome-shell10:15:123
653823996220,25cyclictest1753-21gnome-shell11:31:593
653823996219,26cyclictest1753-21gnome-shell09:13:263
653823996219,25cyclictest1753-21gnome-shell09:46:013
653823996219,25cyclictest1753-21gnome-shell09:46:013
653823996218,26cyclictest1753-21gnome-shell09:30:083
653823996218,26cyclictest1753-21gnome-shell08:56:443
653823996217,27cyclictest1753-21gnome-shell11:35:123
653823996217,26cyclictest737550-21python311:05:323
653823996217,15cyclictest1753-21gnome-shell08:30:163
653823996216,25cyclictest1753-21gnome-shell09:40:133
653823996213,12cyclictest1753-21gnome-shell07:15:343
653823996210,21cyclictest854-21avahi-daemon12:13:113
653823996119,25cyclictest1753-21gnome-shell09:01:443
653823996119,25cyclictest1753-21gnome-shell07:49:243
653823996119,16cyclictest1753-21gnome-shell09:38:383
653823996118,26cyclictest1753-21gnome-shell10:44:403
653823996117,26cyclictest1753-21gnome-shell10:46:413
653823996018,25cyclictest1753-21gnome-shell12:26:243
653823996018,25cyclictest1753-21gnome-shell07:32:383
653823996018,25cyclictest1753-21gnome-shell07:32:383
653823996018,16cyclictest1753-21gnome-shell11:29:233
653823996017,26cyclictest1753-21gnome-shell08:39:393
653823996017,26cyclictest1753-21gnome-shell07:13:513
653823996016,26cyclictest1753-21gnome-shell09:22:273
653823996015,22cyclictest733676-21cat10:55:223
653823995918,15cyclictest1753-21gnome-shell07:20:443
653823995913,16cyclictest686681-21memory08:40:253
653823995818,16cyclictest1753-21gnome-shell11:23:583
653823995812,29cyclictest1753-21gnome-shell09:50:213
653823995712,16cyclictest716426-21ls10:05:243
653823995712,16cyclictest716426-21ls10:05:243
7271962560,13sleep234-21ksoftirqd/210:35:342
653823995619,11cyclictest1753-21gnome-shell08:18:003
653823995617,15cyclictest1753-21gnome-shell11:56:343
653823995615,14cyclictest674576-21cat08:05:233
653816995524,16cyclictest26-21ksoftirqd/109:08:591
65381999511,34cyclictest141rcu_preempt07:55:212
653816995121,6cyclictest26-21ksoftirqd/108:15:101
653811995020,7cyclictest720807-21idleruntime-cro10:20:000
162399500,36rtkit-daemon669654-21switchtime07:50:312
162399500,35rtkit-daemon1753-21gnome-shell08:38:052
7455912490,5sleep02197014-21PK-Backend11:30:140
65381999493,12cyclictest141rcu_preempt11:45:012
162399490,32rtkit-daemon672221-21apt08:00:142
653811994728,15cyclictest13-21ksoftirqd/011:59:260
653811994726,8cyclictest7272002switchtime10:35:350
653816994625,16cyclictest26-21ksoftirqd/109:30:421
653816994624,13cyclictest26-21ksoftirqd/112:25:151
653816994621,20cyclictest26-21ksoftirqd/110:40:011
653816994620,5cyclictest26-21ksoftirqd/107:29:241
653816994619,5cyclictest26-21ksoftirqd/108:40:501
653816994616,5cyclictest26-21ksoftirqd/110:19:421
162399460,34rtkit-daemon1753-21gnome-shell11:00:322
162399460,32rtkit-daemon1753-21gnome-shell11:10:422
162399460,32rtkit-daemon0-21swapper/210:20:372
653819994525,7cyclictest34-21ksoftirqd/207:27:222
653819994523,14cyclictest34-21ksoftirqd/209:30:242
653816994533,8cyclictest769494-21grep12:35:251
653816994524,16cyclictest26-21ksoftirqd/108:05:161
653816994523,14cyclictest26-21ksoftirqd/111:40:011
653816994523,14cyclictest26-21ksoftirqd/111:05:171
653816994523,14cyclictest26-21ksoftirqd/107:15:031
653816994523,13cyclictest26-21ksoftirqd/112:15:021
653816994523,13cyclictest26-21ksoftirqd/112:00:151
653816994523,13cyclictest26-21ksoftirqd/107:50:231
653811994521,15cyclictest13-21ksoftirqd/008:25:130
7008652440,5sleep20-21swapper/209:20:312
653819994426,14cyclictest34-21ksoftirqd/211:50:002
653819994424,15cyclictest34-21ksoftirqd/207:35:312
653819994423,8cyclictest34-21ksoftirqd/209:10:172
653819994421,14cyclictest34-21ksoftirqd/210:50:022
653816994426,6cyclictest26-21ksoftirqd/108:15:001
653816994424,16cyclictest26-21ksoftirqd/108:49:131
653816994424,16cyclictest26-21ksoftirqd/108:20:501
653816994423,16cyclictest26-21ksoftirqd/111:50:001
653816994422,14cyclictest26-21ksoftirqd/110:30:291
653816994422,13cyclictest26-21ksoftirqd/108:50:221
653811994434,6cyclictest13-21ksoftirqd/008:05:530
653811994425,7cyclictest13-21ksoftirqd/010:20:140
653811994425,7cyclictest13-21ksoftirqd/007:40:130
653811994425,7cyclictest13-21ksoftirqd/007:27:140
653811994425,15cyclictest13-21ksoftirqd/012:06:290
653811994420,15cyclictest13-21ksoftirqd/009:40:170
653811994414,25cyclictest704535-31apt-check09:30:300
162399440,5rtkit-daemon0-21swapper/210:28:032
162399440,31rtkit-daemon1753-21gnome-shell11:55:492
162399440,31rtkit-daemon0-21swapper/208:50:312
653819994332,7cyclictest34-21ksoftirqd/209:40:162
653819994324,15cyclictest680605-21cron08:25:002
653819994323,15cyclictest34-21ksoftirqd/207:50:012
653819994320,5cyclictest141rcu_preempt08:17:452
653819994320,14cyclictest34-21ksoftirqd/209:10:012
653819994312,12cyclictest737343-21head11:05:282
653819994311,10cyclictest141rcu_preempt10:50:222
65381699439,31cyclictest251rcuc/109:25:301
653816994325,14cyclictest26-21ksoftirqd/112:20:001
653816994325,14cyclictest26-21ksoftirqd/111:25:281
653816994325,14cyclictest26-21ksoftirqd/110:55:401
653816994325,14cyclictest26-21ksoftirqd/108:01:341
653816994324,6cyclictest26-21ksoftirqd/110:43:271
653816994324,5cyclictest26-21ksoftirqd/108:30:251
653816994324,15cyclictest26-21ksoftirqd/110:02:171
653816994324,15cyclictest26-21ksoftirqd/109:03:241
653816994324,15cyclictest26-21ksoftirqd/107:18:221
653816994323,6cyclictest26-21ksoftirqd/109:10:321
653816994323,13cyclictest26-21ksoftirqd/112:05:221
653816994323,13cyclictest26-21ksoftirqd/107:30:331
653816994323,13cyclictest26-21ksoftirqd/107:30:331
653816994322,6cyclictest26-21ksoftirqd/111:05:001
653816994322,16cyclictest26-21ksoftirqd/110:20:281
653816994320,19cyclictest26-21ksoftirqd/110:05:231
653816994320,19cyclictest26-21ksoftirqd/110:05:231
653816994319,20cyclictest26-21ksoftirqd/109:48:491
653816994319,20cyclictest26-21ksoftirqd/109:48:491
65381699431,18cyclictest1753-21gnome-shell08:35:461
653811994325,6cyclictest13-21ksoftirqd/009:55:010
653811994325,6cyclictest13-21ksoftirqd/007:15:370
653811994325,14cyclictest13-21ksoftirqd/011:30:010
653811994324,6cyclictest13-21ksoftirqd/012:39:340
653811994321,14cyclictest13-21ksoftirqd/009:15:010
653811994321,13cyclictest13-21ksoftirqd/012:20:010
7019292420,6sleep22197014-21PK-Backend09:25:152
653819994226,6cyclictest34-21ksoftirqd/212:35:172
653819994224,6cyclictest34-21ksoftirqd/212:27:472
653819994219,14cyclictest34-21ksoftirqd/212:15:042
653819994214,24cyclictest34-21ksoftirqd/212:30:232
65381999421,36cyclictest438-20systemd-journal11:19:592
653816994230,8cyclictest741078-21grep11:15:331
653816994228,10cyclictest755637-21sshd11:57:441
653816994225,6cyclictest26-21ksoftirqd/107:46:281
653816994224,6cyclictest26-21ksoftirqd/111:50:281
653816994224,6cyclictest26-21ksoftirqd/111:50:281
653816994224,6cyclictest26-21ksoftirqd/108:25:271
653816994224,6cyclictest26-21ksoftirqd/107:20:251
653816994222,6cyclictest26-21ksoftirqd/111:20:151
653816994222,16cyclictest26-21ksoftirqd/107:44:171
653811994225,13cyclictest13-21ksoftirqd/011:20:180
653811994224,6cyclictest13-21ksoftirqd/011:35:520
653811994224,6cyclictest13-21ksoftirqd/010:15:000
653811994224,6cyclictest13-21ksoftirqd/009:36:050
653811994224,6cyclictest13-21ksoftirqd/007:30:160
653811994224,6cyclictest13-21ksoftirqd/007:30:150
653811994223,14cyclictest13-21ksoftirqd/008:24:590
653811994220,11cyclictest13-21ksoftirqd/011:00:010
653819994115,8cyclictest1753-21gnome-shell07:20:202
653819994113,11cyclictest141rcu_preempt07:41:552
65381999411,17cyclictest0-21swapper/209:04:112
65381999411,15cyclictest725215-21strings10:30:242
653816994123,6cyclictest26-21ksoftirqd/112:30:171
653816994123,6cyclictest26-21ksoftirqd/110:10:271
653816994122,6cyclictest26-21ksoftirqd/109:20:261
653816994121,6cyclictest26-21ksoftirqd/108:55:141
653811994124,5cyclictest13-21ksoftirqd/011:55:020
653811994124,5cyclictest13-21ksoftirqd/011:55:010
653811994124,5cyclictest13-21ksoftirqd/009:20:170
653811994124,13cyclictest13-21ksoftirqd/009:05:120
653811994123,7cyclictest13-21ksoftirqd/012:30:240
653811994123,6cyclictest13-21ksoftirqd/010:40:370
653811994123,6cyclictest13-21ksoftirqd/009:45:170
653811994123,6cyclictest13-21ksoftirqd/009:45:170
653811994123,6cyclictest13-21ksoftirqd/008:17:080
653811994122,7cyclictest13-21ksoftirqd/008:14:280
653811994118,14cyclictest13-21ksoftirqd/009:55:020
162399410,30rtkit-daemon0-21swapper/208:30:152
7485542400,7chrt2197014-21PK-Backend11:36:212
65381999404,21cyclictest1753-21gnome-shell09:16:172
653819994022,14cyclictest34-21ksoftirqd/209:40:012
653819994021,14cyclictest34-21ksoftirqd/209:58:272
653819994021,14cyclictest34-21ksoftirqd/208:47:482
653819994012,12cyclictest141rcu_preempt10:05:442
653819994012,12cyclictest141rcu_preempt10:05:442
653816994023,6cyclictest26-21ksoftirqd/112:20:331
653816994023,6cyclictest26-21ksoftirqd/109:36:511
653816994023,5cyclictest26-21ksoftirqd/110:00:031
653816994023,5cyclictest26-21ksoftirqd/107:55:431
653816994022,6cyclictest26-21ksoftirqd/111:40:231
653816994022,6cyclictest26-21ksoftirqd/111:10:181
653816994022,6cyclictest26-21ksoftirqd/110:45:121
653816994022,6cyclictest26-21ksoftirqd/109:40:341
653816994022,6cyclictest26-21ksoftirqd/109:17:331
653816994020,6cyclictest26-21ksoftirqd/110:25:271
653811994025,12cyclictest655496-21awk07:15:000
653811994023,6cyclictest13-21ksoftirqd/012:05:000
653811994023,6cyclictest13-21ksoftirqd/008:50:020
653811994022,6cyclictest13-21ksoftirqd/007:40:020
653811994022,14cyclictest714722-21memory10:00:230
653811994022,13cyclictest13-21ksoftirqd/007:50:290
653811994019,12cyclictest13-21ksoftirqd/009:25:130
653811994018,13cyclictest13-21ksoftirqd/011:45:020
653811994018,13cyclictest13-21ksoftirqd/008:00:210
653811994017,14cyclictest13-21ksoftirqd/011:00:220
653819993922,13cyclictest34-21ksoftirqd/207:10:152
653819993921,14cyclictest34-21ksoftirqd/210:12:512
653819993920,14cyclictest34-21ksoftirqd/211:20:442
653819993910,11cyclictest141rcu_preempt10:01:002
653816993924,10cyclictest747133-21gpgv11:35:021
653816993922,6cyclictest26-21ksoftirqd/107:39:001
65381199392,7cyclictest765120-21turbostat12:25:000
653811993923,12cyclictest13-21ksoftirqd/009:00:010
653811993921,6cyclictest13-21ksoftirqd/008:36:370
653811993920,6cyclictest13-21ksoftirqd/010:05:080
653811993920,6cyclictest13-21ksoftirqd/010:05:080
653811993920,14cyclictest13-21ksoftirqd/007:55:150
653811993920,12cyclictest13-21ksoftirqd/010:31:110
653811993918,17cyclictest13-21ksoftirqd/011:09:580
653811993916,14cyclictest13-21ksoftirqd/009:03:330
162399390,28rtkit-daemon0-21swapper/211:51:052
162399390,28rtkit-daemon0-21swapper/211:51:042
162399390,26rtkit-daemon2197014-21PK-Backend12:05:582
7284292380,15sleep2341ksoftirqd/210:40:222
65381999388,16cyclictest686292-21cstates08:40:172
65381999386,16cyclictest141rcu_preempt10:15:222
65381999382,14cyclictest854-21avahi-daemon08:10:002
653819993821,13cyclictest757186-21users12:00:332
653819993820,14cyclictest34-21ksoftirqd/212:25:022
653819993820,14cyclictest34-21ksoftirqd/210:58:232
653819993819,14cyclictest34-21ksoftirqd/209:50:272
653819993819,14cyclictest34-21ksoftirqd/208:25:112
653819993816,13cyclictest34-21ksoftirqd/207:15:232
653816993819,6cyclictest26-21ksoftirqd/109:50:181
653811993821,13cyclictest760464-21memory12:10:240
653811993820,6cyclictest13-21ksoftirqd/011:45:180
653811993819,6cyclictest13-21ksoftirqd/010:25:360
653811993819,15cyclictest13-21ksoftirqd/007:21:340
653811993816,14cyclictest13-21ksoftirqd/008:48:400
653811993816,13cyclictest13-21ksoftirqd/008:32:400
653811993815,14cyclictest13-21ksoftirqd/011:10:110
65381999372,8cyclictest1753-21gnome-shell12:14:542
65381999372,19cyclictest1753-21gnome-shell11:30:162
653819993720,12cyclictest743652-21apt11:25:142
653819993719,14cyclictest34-21ksoftirqd/209:45:242
653819993719,14cyclictest34-21ksoftirqd/209:45:242
653819993719,13cyclictest34-21ksoftirqd/207:30:222
653819993719,13cyclictest34-21ksoftirqd/207:30:212
65381999371,32cyclictest676165-21runrttasks08:10:212
65381199376,5cyclictest141rcu_preempt12:25:230
653811993716,12cyclictest13-21ksoftirqd/007:50:010
65381199362,6cyclictest141rcu_preempt10:52:010
653811993511,9cyclictest141rcu_preempt10:46:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional